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Introduction 

The prime geodesic theorem (PGT) gives a growth asymptotic for the number of closed geodesies counted 
by their lengths [15. 21, 27, 28, 30, 31, 33, 43]. Classically, it was known for manifolds of strictly 
negative sectional curvature. In [1 1] , one finds a generalization to higher rank locally symmetric manifolds, 
counting only regular geodesies. In [13] the PGT is extended to non-regular geodesies, under the proviso 
of a geometric regularity condition on the manifold. 

The current paper is dedicated to quotients of the symmetric space of SL4(]R). In this situation the 
regularity condition fails. As a consequence, the analytical difficulties grow exponentially. The group 
SL4(M) is the "biggest" semisimple group for which the unitary dual is known. This enables us to give 
prime geodesic theorems with remainder term. 

In [9, 12, 11, 13], the prime geodesic theorem is applied to derive class number asymptotics. The last 
open case for these is the case of complex quartic fields for which the PGT required is the one of SL4(M). 
As there are additional algebraic obstacles, this application of the PGT is not straightforward and will 

be subject of a subsequent paper. 

In what follows we give a summary of the arguments and techniques used in the proof of our main results. 
In particular we shall point out the difficulties that arise in applying the methods of [11] to our situation. 
Section 1 provides some necessary background and preliminary results. In Section 2 we introduce the 
zeta functions we shall be studying and prove some of their analytic properties. The results of Section 
1 are made use of here in the definitions of the zeta functions and proofs of their analytic properties. In 
Section 3 we apply standard techniques from analytic number theory to the results of Section 2 in order 
to prove the Prime Geodesic Theorem required in our context. 

Our strategy is that we define a suitable zeta function and use the trace formula with special test- 
functions to derive analytic continuation as in [8] , so that we can use the techniques of the proof of the 
prime number theorem. It is in the analytic continuation that we encounter the first obstacle. We say 
that an element <? S G is weakly neat if the adjoint Ad{g) has no non-trivial roots of unity as eigenvalues. 
A subgroup of G is weakly neat if every element is. The results of [8] use the assumptions that the group 
G has trivial centre and the group F is weakly neat. Note that, if G has trivial centre then F weakly 
neat implies F torsion free, since any non weakly neat torsion elements must be central. To generalise 
the results of [8] to the case where F is not weakly neat requires two things. Firstly, we need to modify 
the definition of the zeta function from that given in [8]. Secondly, the definition of the zeta function 
includes the first higher Euler characteristic of the spaces Xp^ for 7 e <^(F). In [8] these are only defined 
for torsion free so we need to broaden the definition. 

In Section 1 we give a definition of first higher Euler characteristics which is general enough for our 
application. We further prove that its value is always positive in the cases we consider. This fact is 
needed in the proof of the Prime Geodesic Theorem. In [8] it is shown that the position of the poles and 
zeros of the generalised Selberg zeta function depend on the Lie algebra cohomology of the irreducible, 
unitary representations of SL4(IR). Using a result of Hecht and Schmid ([20]) it suffices to look at the 
infinitesimal characters of the irreducible, unitary representations of SL4(M). For this purpose, in Section 
1 wc also describe the imitary dual of SL4(M) using a result of Speh ([39]) and give a result about the 
infinitesimal characters of certain elements of this set. 

In Section 2 we define the generalised Selberg zeta function and use the trace formula to deduce its analytic 
properties. In particular, we give a formula for its vanishing order at a given point and prove a functional 
equation, from which we can deduce that it is of finite order. We then define the generalised Ruelle zeta 
function and prove that it is a finite quotient of generalised Selberg zeta functions. In particular, in the 
cases we are interested in it has a zero at s = 1 and all other poles and zeros in the half plane Res < |, 
and is furthermore of finite order. We introduce the Ruelle zeta function as its logarithmic derivative is 
a Dirichlet series from whose properties we can prove the Prime Geodesic Theorem. 

The definitions of the generalised Selberg and Ruelle zeta functions depend on the choice of a finite 
dimensional virtual representation of a particular closed subgroup M of SL4(R). The trace of this 
representation at elements of F appears in the Dirichlet series arising as the logarithmic derivative of the 
Ruelle zeta function. We can choose such a representation a so that its trace is zero for all non-regular 
elements of F. 

In Section 3 we apply the methods of [34] together with standard techniques of analytic number theory 
to prove a Prime Geodesic Theorem. In carrying out the application to class numbers it is necessary 
to show that certain subsets of the summands contribute negligibly to the asymptotic. This is done 
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using a "sandwiching" argument. To carry out this "sandwiching" argument we define a sequence of 
Dirichlet series, which are not connected to any zeta function, but whose analytic properties can also be 
deduced from the Selberg trace formula. We use a version of the Wiener-Ikehara theorem to prove an 
asymptotic result for an increasing sequence of functions derived from these Dirichlet series, which we 
use to "sandwich" the product over the elements we are interested in against the sum over all elements 
which comes from the Ruelle zeta function. Unfortunately the asymptotics we are able to derive from 
the Wiencr-Ikchara theorem do not provide an error term like those wc can deduce from the Ruelle zeta 
function using the methods of [34]. This is why we lose the error term in Theorem 3.2. 

We finish this introduction with a few remarks about the limitations of the method used here in terms 
of further applications and mention a couple of other recent results in the same direction. In order to be 
able to make use of the trace formula for compact spaces we have had to limit our sum over class numbers 
by means of the choice of a finite set of primes, as described above. In [12] Deitmar and Hoffmann have 
been able to use a different trace formula to prove that as a; — > oo 



where the sum is over all isomorphism classes of orders in complex cubic fields. 

In order to get the error term in the Prime Geodesic Theorem we have made use of the classification of 
the unitary dual of SL4(M). At present the unitary dual is not known for any higher dimensional groups 
so a Prime Geodesic Theorem with error term is not possible using our methods. Finally wo mention 
that the correspondence between geodesies and orders actually works by identifying primitive geodesies 
with fimdamental units in orders. By Dirichlet's unit theorem, an order in a number field F has a unique 
fundamental unit (up to inversion and multiplication by a root of unity) only if F is real quadratic, 
complex cubic or totally complex quartic. Hence an asymptotic of our form can be proven only in these 
three cases. 




R{0)<x 
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1 Euler Characteristics and Infinitesimal Characters 



In this section we introduce some concepts and prove some results which will be needed for our consid- 
eration of the zeta functions in the next section. 



1.1 The unitary duals of SL2(M) and SL4(M) 

The unitary dual G of a locally compact group G is the set of all equivalence classes of irreducible unitary 
representations of G. If tt is an irreducible, unitary representation of G then we shall write, by slight 
abuse of notation, n G G. Let Gi = SL2(K). We define the following subgroups: let Ki be the maximal 
compact subgroup S0(2); let Mi = {±72}- let Ai be the group of diagonal matrices diag(a, a""'^ where 
a > 0, and let A''i be the group of upper triangular matrices in Gi with ones on the diagonal. Let fli, 4i, 
Oi, ni be the respective complexified Lie algebras. Let pi e a| be defined by p{dia.g{a, —a)) = a, and let 
Pi = MiAiNi be the parabolic subgroup of Gi with split torus Ai and unipotent radical Ni. We will 
use the notation of [26], Chap. II. For n > 2 we have the discrete series representations ^+ and ^~ of 
Gi . We also denote the two limit of discrete series representations by and . 

Let r+,T~ be the two characters of M - 1. For u G al we define ^^'^ = lnd$^{T^ O i^). For a; e M 
define the principal series representations 

Then the representations .fj^^-''-'^- are imitary and are all irreducible except for = Q)^^ . The only 

equivalences among the representations 3^^'"^ are that ^+'*^ and ^--^^ are equivalent to ^+'-" and 
(^--ix respectively for all a; € K. We also define the complementary series representations = ^+'^p^ , 
for < a; < 1. We have the following classification theorem: 

Theorem 1.1 The unitary dual o/SL2(M) consists of 

a) the trivial representation; 

b) the principal series representations ^+'''^ for a; e K and ^->»^ for a; e K \ {0}; 

c) the complementary series for < a: < 1; 

d,) the discrete series Si^ and, 9^ for n > 2 and limits of discrete series S^f and, S^i ■ 
The only equivalences among these representations are ^ (^±-ix y^^, a; G M. 



Proof: [26], Theorem 16.3. □ 

Now let G = SL4(]R) and K = SO (4), so K is & maximal compact subgroup of G. Let P' = M'A'N' 

be a parabolic subgroup of G with split component A' and unipotent radical N' . Let g and a' be the 
complexified Lie algebras of G and A' and let a'* be the complex dual of a'. Let p' be the half sum of the 
positive roots of the system (g', a'). Then we can define induced representations in an entirely analogous 
way to that used for for Gi = SL2(IR) above. For an irreducible, unitary representation t of M' and 
for V e a'* we write the corresponding induced representation of G as Ind p/(r (g) i^). li v = ixp' for 
some a; e M then the induced representation Ind p/ (r ® v) is unitary with respect to the inner product 
(./j.?) = /fi-(/('^)' t^i^ '^^^'^ Ind p/(t(8) v) a principal series representation. For 

other choices \s v G a'* it may be possible to make Ind p, (r (g) v) unitary with respect to a different inner 
product, in which case we call Ind '^i{t®v) a complementary series representation. There are also certain 
irreducible, unitary subrepresentations of induced representations (sec [39], pl21) which are called limit 
of complementary series representations. These limit of complementary series representations can often 
be realised as induced representations from a parabolic subgroup P" D P' . 

We define the following subgroups of G. Let 

SL^' 



M 



-'2 K'''^) 

SL^(M 
{x,y) G Mat2(M) x Mat2 



det (a;),det (y) = ±1 
det (a;)det (y) = 1 



Let A be the group of all diagonal matrices diag(a, a, a where a > 0. Finally, let N = 

( I2 Mat2(l 



/, 
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. Let P be the parabolic subgroup of G with Langlands decomposition P — MAN. For 
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rrii, m2 G N, wc denote by TTmi.m2 the representation of M induced from the representation (S f^^2 
of SL2(R) X SL2(M). For m gN, let Tf^ = Tf^.m and let = Ind p (tt^ 'Si ^Pp)- The representations 
each have a unique irreducible quotient known as the Langlands quotient (see [26], Theorem 7.24). We 
denote the Langlands quotient of by tt^. 

We have the following classification theorem: 

Theorem 1.2 The unitary dual o/SL4(M) consists of 

a) the trivial representation; 

b) principal series representations; 

c) complementary series representations Indp (7fm tpp), for m e N and Q <t <\; 

d) complementary series representations induced from parabolics other than P = MAN; 

e) limit of complementary series representations, which are irreducible unitary subrepresentations of Im, 
for m e N; 

f) the family of representations -Km, indexed by m gN. 

Proof: This follows from [39], Theorem 5.1, where the unitary dual of SLJ(R) = {X € Mat4(M) : 
det X = ±1} is given. □ 

1.2 Euler chciracteristics 

Let G be a real reductive group and suppose that there is a finite subgroup E of the centre of G and 
a reductive and Zariski-connected linear group such that G/E is isomorphic to a subgroup of (R) 
of finite index. Note these conditions are satisfied whenever G is a Levi component of a connected 
semisimple group with finite centre. Let K he a maximal compact subgroup of G and let F be a discrete, 
cocompact subgroup of G. Let Xr be the locally symmetric space r\G/K. If F is torsion-free we define 
the first higher Euler characteristic of P to be 



where h^{Xr) is the jth Betti number of Xr- We want to generalise this definition to cases where 
the group T is not necessarily torsion free. We prove below a proposition which allows us broaden the 
definition to all cases required by our applications. 

Let 6 be the Cartan involution fixing K pointwise. There exists a 0-stable Cartan subgroup H = AB of 
G, where A is a connected split torus and B d K is & Cartan of K. We assume that A is central in G. 
Let C denote the centre of G. Then C C H and we write Be, Tc for i? n C and F n C respectively. Let 
G^ be the derived group of G and let F^ = fl FG. We note in particular that, since G = G^G, we 
have F c F^G c T^AB. Let F^ = ^ n TcBc be the projection of Tc to A. Then F^ is discrete and 
cocompact in A (see [42], Lemma 3.3). 

Let flR be the real Lie algebra Lie(G) with polar decomposition 0k = 4m © pR- Let 6 be a fixed non- 
degenerate invariant bilinear form on the Lie algebra which is negative definite on and positive 
definite on pR. Then the form —b{X,6{Y)) is positive definite and thus defines a left invariant metric 
on G. Unless otherwise stated all Haar measures will be normalised according to the Harish- Chandra 
normalisation given in [19], §7. Note that this normalisation depends on the choice of the bilinear form b 
on g. On the space G/i? we have a pseudo-Riemannian structure given by the form b. The Gauss-Bonnet 
construction generalises to pseudo-Riemannian structures to give an Euler-Poincare measure r] on G/H. 
Define a (signed) Haar measure on G by 



The Weyl group W = W{G, H) is defined to be the quotient of the normaliser of _ff in G by the centraliser. 
It is a finite group generated by elements of order two. We define the generic Euler characteristic by 



dim Xr 



Xi(Xr) = xi(F) 



^ i-iy+'jh^iXr) 



IJ'EP = (Haar measure on H). 



Xgen(r) = Xgen(^r) = 



PEP{T\G) 

\W\ 
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Proposition 1.3 Assume T is torsion free, A is central in G of dimension one and V G T is of finite 
index in T. Then A/Ta acts freely on Xr and Xgen(-^r) = Xi(r)vol(y4./r^). It follows that 



xi(r) = xi(r') 



[r : r'] 



Proof: The group Ar = A/Fa acts on r\G/B by multiplication from the right. We claim that this 
action is free, i.e., that it defines a fibre bundle 

Ar r\G/B T\G/H. 

To see this let TxaB = TxB for some a € A and x & G. Then a = x~^jxb for some 7 G F and b £ B. 
Since T c T^C C T^AB we can write 7 as j'^a^b^, with 7^ e and G A and b^ € Be = B nC. It 
follows that £ T a- Since A is central in G, wc can write = aa~^xh~^x~^h~^ . Since G G^ and 
aa~^ G A c C, we must have aa~^ = 1, so a = Uj £ Ta, which implies the claim. In the same way we 
see that we get a fibre bundle 

Ar T\G/K AT\G/K. (1) 

Let X be the usual Eulcr characteristic. From [24] we take the equation x(K/B) = \W\. Using multi- 
pHcativity of Euler numbers in the fibre bundle K/B AT\G/B AT\G/K we get 



= yol{A/TA) 
= vol(A/r^) 



\W\ 

x{r\G/H) 

X{K/B) 

x{Ar\G/B) 

X{K/B) 



= yo\{A/Ta)x{AT\G/K) 
= yo\{A/TA)x{A\XT). 

It remains to show that x(A\Xr) = Xi(r)- 

Let aR and be the Lie algebras of A and respectively. Then we can write 

0R = aR ® Or ® 3r, (2) 

where 3k is central in flg. Let X be the bi-invariant vector field on r\G/K generating the action. 
Then we can consider X as an element of Or under the decomposition (2). Considering the dual of (2), 
we can identify aj^ with a subspace of g^. Since a is central in g, a non-zero element of gives us an 
Ar-invariant, closed 1-form u) on T\G/K such that for every TgK e T\G/K we have ujiTgK){X) ^ 0. 
Since Ay- = M/Z is connected and compact, the cohomology of the de Rham complex T\G/K coincides 
with the cohomology of the subcomplex of ^r-invariants Q.{r\G / K)-^^ . Using local triviality of the 
bundle one sees that 

n(r\G/jr)^^ = 'k*q.{at\g/k) ® ■k*q.{av\g/k) a w, 

where tt* denotes the projection map. Set Go = ■k*Q.{AT\G / K) and Gi = Go ® Go A a; = Q.{T\G/Ky^. 
Then 

H'P{Ci)=HP{Co)®H'p-\Cq) 

and so 

Xi(Ci) = ^(-l)f+VdimirJ'(Gi) 

p>0 

= (dimfl-P(Go) + dimiff-i(Go)) 

= ^(-l)f+i(p-(p+l))dimirf(Go) 

p>0 

= ^(-lfdimFP(Go) = x{Co). 

p>0 
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This gives the required result. □ 

Let r be a discrete, cocompact subgroup of G. Then F has a torsion-free subgroup V of finite index. 
Suppose that the torus A is central in G and of dimension one. Define Ta and as above. 

We define the first higher Euler characteristic of F as 

XI (Xr) = XI (r) = XI (r')^^^-^. (3) 

Propostition 1.3 shows that this is well-defined. We note that in the case that F itself is torsion free, this 
definition of first higher Euler characteristic agrees with the one given above. 



1.3 Euler- Poincare functions 

For the next definition we assume G to be a semisimple real reductive group of inner type and we fix 

a maximal compact subgroup K . Wc further assume that G contains a compact Cartan subgroup. Let 
0R be the real Lie algebra Lie(G) with polar decomposition 0r = 4r © pR, and write g = t © p for its 
complexification. Recall that we are using Harish-Chandra's Haar measure normalisation as given in 
[19], §7 and this normalisation depends on the choice of an invariant bilinear form h on fl, which for our 

purposes in this section wo shall leave arbitrary. 

A virtual representation ct of a group is a formal difference of two representations a — — cr^, which 
is called finite dimensional if both and a~ are. Two virtual representations a — — a~ and 
r = r+ — r~ of a group are said to be isomorphic if there is an isomorphism cr+ © t~ = t+ © it~ . The 
trace and determinant of a virtual representation cr = cr+ — cr~ are defined by tr u = tr cr+ — tr a~ and 
det G = det 0-+/det a~. The dimension of a is defined as dim a = dimcr+ — dim a". 

If is a representation space with Z-grading then wc shall consider it naturally as a virtual representation 
space by = 14ven and V~ = Kdd- In particular, if y is a subspace of q we shall always consider the 
exterior product /\* V as a. virtual representation f\*V= V — /\°'^'^ V with respect to the adjoint 

representation. We consider symmetric powers and cohomology spaces similarly. 

For a smooth function / on G of compact support and an irreducible unitary representation (tt, K-) € G 
define the operator 

Jg 

on Vtt- Since / is smooth and has compact support, 7r(/) is of trace class. 

Let (cr, Va) he a finite dimensional virtual representation of G. An Euler-Poincare function fa- for u is a 
compactly supported, smooth function on G such that f^ (kxk~^) = f^ {x) for all fc € and for every 
irreducible unitary representation (tt, V^) of G the following identity holds: 

dim(p) ^ 

tr TT (/<,) = (-1)" dim {v^ ® f^p^V,) , (4) 

p=0 

where the superscript K denotes the subspace of K invariants. By [29] such functions exist. We note that 
the value of such functions depends on the choice of Haar mcasiirc. Wo shall assume all Eulcr-Poincare 
functions are given with respect to the Harish-Chandra normalisation. In the following lemmas we prove 
some of their properties. 

Lemma 1.4 Let G denote a semisimple real reductive group of inner type, with connected component 
G°, maximal compact subgroup K and compact Cartan subgroup T c K. Let G+ = TG°. Further let a 
be a finite dimensional representation of G, = a\G+ and f^ an Euler-Poincare function for a on G. 

Then fa\G+ an Euler-Poincare function for cr+ on G+. 

Proof: This is Lemma 1.5 of [8]. □ 

Lemma 1.5 Let H, Hi, H2 be real reductive groups of inner type such that H = Hi x H2. Let a be an 
irreducible representation of H. There exist irreducible representations ai,a2 of Hi,H2 respectively such 
that (7 = (Ti (g) (T2 and let f^ be an Euler-Poincare function for Oi on Hi . 

Then fa{hi,h2) = fuiihijfa^^ihi) is an Euler-Poincare function for a on H. 
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Proof: This is a straightforward computation. □ 

Lemma 1.6 Let G denote a semisimple real reductive group of inner type, with maximal compact sub- 
group K and compact Cartan subgroup T C K . Let g G G be central. Let a be a finite dimensional 
representation of G and let f^, ha- be Euler-Poincare functions for a on G. Then fcr{g) = hc{g). 

Proof: Since g is central, the orbital integral can be written as 

Og{fa)= [ fa{x-'gx)dx = fa{g), 

JGg\G 

where Gg denotes the centraliser of gi in G (which in this case, since g is central, is the whole of G). 

Similarly Og{ha) = ha(g). By [8], Proposition 1.4, the value of the orbital integral Og{fa) of an Euler- 
Poincare function for cr on G depends only on g, not on the particular Euler-Poincare function chosen. 
Hence, fcr{g) = Og{fa) = Og{ha) = hdg), as claimed. □ 

Lemma 1.7 Letgi be an Euler-Poincare function for the trivial representation on S1j2{^). Then gi{l) = 

Proof: For u e M let ^+'*'' and be principal series representations on SL2(M). For n G N, n > 2 

let ^+ and S>~ be discrete series representations on SL2(M) and write for ^+©^~. By [26], Theorem 
11.6 there is a constant M > such that for any compactly supported, smooth function / on SL2(M) we 
have 



/(I) = M£(n- l)tr^±(/) 

+ ^ j tr ^+''''(/)t;tanh + tr ^-■'''(/)t;coth dv. 

Lemma 1.3 of [8] tells us that tr ^^'"^^{gx) = for all ii G M, so we have 

oo 

ffi(l)=M^(n-l)tr^±(ffi). (5) 



n=2 

By the definition of an Euler-Poincare function, for all n > 2 



2 



-U NT^ / i aP \S0(2) 

tr^±(5i) = ^(-If dim ® /\ p) 

p=0 



where p is the complex Lie algebra 



p = <l( I \ ]:a,beC 



Hence gi(l) G M. 

We want to know for which values of n the trace tr &^{gi) is non-zero. For this we need to know the 
S0(2)-types of 9^ and p. 

For I e Z, define the one dimensional representation e; of SO (2) by 

eiR{e) = where RiQ) = f ''"^f ^ G S0(2). 

^ ' \ smB cosO J ^ ' 

Note that eq is the trivial representation, which we shall denote by triv. 

The unitary dual of S0(2) is the set {e; : Z G Z}. 

We have the following isomorphisms of SO(2)-modules: 

A A 1 A 2 

P\ P = triv, /\ p = £2 ® £-2, /\ P = triv. 
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Lemma 1.8 For n e N we have an isomorphism of SO {2) -modules: 

j = n mod2 

Proof: Let r„ be the unique n-dimensional representation of SL2(IR) (see [26], Chapter II §1). It follows 
from the definition of t„ that the following isomorphism of SO(2)-modules holds: 

Ui<n-1 
j = „_l mod(2) 

Let Pi = MiAiNi be the minimal parabolic of SL2(IR.). Then the unitary dual of Mi = {1, —1} consists 
of two one dimensional representations, which we denote by 1 = triv and —1. We denote by pi the 
character of Ai 

^^(" a-' 

and write 7r±i^„_i for Ind p±l(g)(n— l)pp. Then we have the following exact sequences of SL2(R)-modules 
(see [26], Chapter II §5): 

—^ — > 7ri,„_i — > Tn-i — » 0, H gN, H even 

and 

^ ^ 7r_i^„_i Tn-i ^0, n e N, n odd, nj^l 
and the isomorphism of SL2 (M)-modules 

= ^-1,0- 

Prom the so-called compact picture of induced representations given in [26], Chapter VII §1, it is fairly 
straightforward to compute the following isomorphisms of SO(2)-modules, where the sums are over all 
integers with the same parity: 

TTl.Ti-l = Sj, 7r_l,„_l = Ej. 

j even j odd 

The lemma then follows easily by combining the various elements of the proof. □ 

From the previous two lemmas we can see that tvS>^{gi) is non-zero only if n = 2 and in that case 

tr^2'^(.gi) = -2, so ,gi(l) -2M. 

It remains to show that (?i(— 1) = ffi(l)- Let be the right multiplication operator of SL2(IR) on the 
space (SL2(M)) of smooth, compactly supported functions on SL2(IR). That is, Rzg{x) = g{xz) for 
all X, z G SL2(M) and g G (SL2(R)). Let tt be an irreducible, unitary representation of SL2(M). 
The matrix —1 = —I2 is central in SL2(Ili) so 7r(— 1) commutes with it(x) for all x G SL2(IR) and 
hence, by Schur's Lemma ([26], Proposition 1.5) is scalar. This means that for any irreducible, unitary 
representation tt on SL2(R) we have tr tt (ii-igi) = ■7r(— l)tr 7r(g'i). Thus we get 
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(-1) = i?_i<?i(l) = M^(n-l)^±(-l)tr^±(5i) = &t{-l)gi{l), 



n=2 



where ^^(—1) is a scalar, which we now compute. For g G and 2; G C we have the action 



^til 5 ]g{z) = {-hz + d)-^g 



Hence 



Similarly we get 



^2+( ^ _i )g{z) = {-l)-'gi-^^]=g{z). 



^2-( ^ _i ]9{z)=9{z) 



so ^2 (— 1) = 1 and the lemma is proved. □ 
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1.4 Euler chciracteristics in the case of SL4 



We now consider the particular case G = SL4(R). Let K — SO (4), a maximal compact subgroup of G 
and let F be a discrete, cocompact subgroup of G. Let A be the rank one torus 



( / a 



A= ( 



and let B be the compact group 



B is a compact Cartan subgroup of 
M = 



Let 



and let P 
Let 



B 



S0(2) 



S0(2) 



: a > 



SL±( 



. Mat.(R) X Mat.(M)| "^Jfi^^^^ 



N : 



I2 Mat2( 
I2 



MAN be a parabolic subgroup of G with Levi component MA and unipotent radical A''. 



\ 



< a < 1 



be the negative Weyl chamber in A with respect to the root system given by the choice of parabolic. Let 
(fp(r) be the set of F-conjugacy classes of elements 7 € F which are conjugate in G to an element of 
A~B. We shall use this notation for the rest of this section. 

Wc say g G G is regular if its ccntraliscr is a torus and non-regular (or singular) otherwise. Clearly, 
for 7 G F regularity is a property of the F-conjugacy class [7]. Let [7] G (?p(F) and write Gj for the 
centraliser of 7 in G. The element 7 is conjugate in G to an element a^b^ e A~B and we define the 
length Ij of 7 to be /-y = 6(loga-y, loga-y)^/^. It follows that if 7 is regular then G-y = AB. 

Let be a maximal compact subgroup of G-y, containing B. Then the group i? is a Cartan subgroup of 
K^, the product AB is a 0-stable Cartan subgroup of G^ and A is central in G^. If we let = T HG^ 
denote the centraliser of 7 in F then F^ is discrete and cocompact in G-y. Let F' be a torsion free subgroup 

of finite index in F and let F^^^, = F' n G,y. Then F!!^ is a torsion free subgroup of finite index in F-^,. We 



define T^^a, F' ^ as in Section 1.2. The first higher Euler characteristic Xii^^y) of ^■y is then defined as 



in (3) as: 



xi(r^) 



[F,:Fg 



xi(r;). 



(6) 



Lemma 1.9 For 7 e ^p(F) we have that 



C-yVOl{T.y\G.y) 



70 



where G-y is an explicit constant depending only on 7, which is equal to one when 7 is regular, 
the primitive geodesic underlying 7. 



70 IS 



Proof: Since F' is torsion free we may take from the second proposition in section 2.4 of [7] the equation 



G,yVOl(F;\G-y) 

A' 



where A^^, denotes the volume of the maximal compact flat in F'\G/-ftr containing 7, and Cy is an explicit 
constant depending only on 7. We note that in [7] there is an extra factor in the equation which does not 
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show up here. The reason is that in [7] a differential form, not a measure, was constructed. The vahic of 
the constant Cj is given in [7] in terms of the root system of with respect to a Cartan subgroup. In 
the case that 7 is regular, as noted above we have that G-y = AB and hence G-y is a Cartan subgroup of 
itself. It is then easy to see that value of Cy in this case is one. 

We denote by the volume of the maximal compact flat in T\G/K containing 7. The values of A-y and 
are given by the volumes of the images of T^^aXA and T'^^^A respectively under their embeddings 
into T\G/K and T'\G/K respectively. Since A is one-dimensional, A-y = l^^. Furthermore, from the 
definition of the groups F^^a and F^^^ it follows that 

\P'i,A ■ ^''y,A\ = ^yl\ = ^'■y/ljo- 

Since 

vo1(F;\G^) = [F^ : F;] vo1(F^\G^), 
the lemma follows from (6). □ 



Proposition 1.10 For all 7 G <^p(r) the Euler characteristic Xi(r^) is positive. If j is regular then 

[Ty,A ■■ T'^^^] 



xi(r^) = 



[F,:r; 



7J 



Proof: If 7 is regular then Gy = AB, Ky = B and F^ is a complete lattice in Gy so Xr' = T'^\Gy/ = 
S^, the unit circle in C. The Betti numbers h^{S^) are equal to zero for j ^ 0,1 and one for j = 0, 1, 
hence Xi(r^) = 1. The claimed value for xi(r-y) then follows immediately from (6). 

Suppose 7 is not regular. Since 7 e <^p(F), we have that 7 is conjugate in G to a-yh-y G A~B c AM. 
Let ((T, Va) be a finite dimensional representation of M and let Km be the maximal compact siibgroup 
S(0(2) X 0(2)) of M, which contains the compact Cartan B of M. We saw above that there exist Euler- 
Poincare functions of a on M. Let f^ be one such. We denote by My the centraliser of by in M and by 
O^ifa) the orbital integral 



/ fa {xbjX ^) dx. 

Jm/m^ 



IM/M., 

From Proposition 1.4 of [8] we get the equation 

Ot'^{fa)=C^tTa{by), 

where Cy is the constant from Lemma 1.9. Together with Lemma 1.9 this gives us 

_ 0,^(^)vol(F^\G^) 
^'^^^ ly,tva{b^) 

Choosing a = 1, the trivial representation of M, this simplifies to 

x.(r,) = "'''^-'7"^-^°-' . (7) 

'70 



To complete the proof of the theorem we shall show that the orbital integral {fi) is positive. In the 
case we are considering 




where 1 denotes the identity matrix in SL2(IR), hence it is central and My = M so we have simply that 

0^^{f^) = fM. (8) 

The group M ^ SL2(M) x SL2(M) is the connected component of M ^ S(SL^(M) x SL^(M)). M has a 
maximal compact subgroup Km = 0(2) x 0(2) and compact Cartan Tm — SO (2) x SO (2). We have that 

TmM = M . Hence by Lemma 1.4, since fi is an Euler-Poincare function for the trivial representation 
on M we have that /i = is an Euler-Poincare function for the trivial representation on M. 
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Let 9i, hi be Euler-Poincarc functions of the trivial representation on SL2(ffi.). By Lemma 1.5 the function 
fi = gihi is an Euler-Poincare function of the trivial representation on M. 

We recall that 

±1 

±1 



b,= 



which is central in M and deduce from Lemma 1.6 that 

/i(^) = Mbj) = ~h{h^) = 5i(±l)/ii(±l). 
Prom Lemmas 1.6 and 1.7 it follows that 

ffi(i) = ffi(-i) = /ii(i) = /ii(-i)eR 

and so fi{b^) is positive. 
1.5 The unitary dual of K 



□ 



M 

Let Km = if fl M = S(0(2) x 0(2)). We shall need in the proof of later results to know the unitary dual 
Km of Km, which is given in the following proposition. 

First we define the following one dimensional representations of SO (2) and SO (2) x SO (2). 

eiR{e) = e''^ for all I e Z, 



'''' ( ^^^^ R{rj) ) = ^ ^ ^' 



where 



R(e) = ( '^^^^ ) e so(2). 

Note that eq and eo,o ai"© the trivial representation of their respective groups. 

Proposition 1.11 For l,k ^'L not both zero there are unique representations 6i^k of Km on with 

^;,fe|sO(2)xSO(2) = £i,fc ffi S-l,-k, 

and 



I -1 



V 



1 / 



(2:1, 2^2) = {zi^Zy). 



We can also define the representation 6 of Km on C by 

6{X,Y){z) = (detX)^ = (detF)^. 

We have that Km = {triv, d, 5i^k ■ l,k € not both zero}. 

Proof: We have seen that SO (2) = {si : I G Z}. In general, for two locally compact groups H and K, 

the map (ct, r) i-^- a (8) r defines an isomorphism H x K = H x K. Thus the map from SO (2) x SO (2) to 

S0(2) X S0(2) given by {ti,T2) i-^ n ig) r2 is an isomorphism. Hence we have that S0(2) x S0(2) is the 
set 



{ei O £fe| Z, e Z} = {e;,fe| l,k€ Z}. 



Let 



/ -1 



T = 



1 / 



and 



m 
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For l,k GZwe note that 

Si,k{T)Si,k{T){zi,Z2) = {zi,Z2)=Si,k{T^){zi,Z2) 

and 

SiMmAR{0,v))SiAT){zi,Z2) = (e-^('^+'='')zi,e^('^+'='')z2) 

= SiiRi-e,-r,)){zi,Z2) 
= 5i{TR{e,r,)T){zuZ2), 

so 5i^k is indeed a representation. We shall show that the representations given in the proposition are in 
fact the only irreducible unitary representations of Km- 

Let T G K]\[. Then by [26], Theorem lT2(d) the representation r restricted to SO (2) x SO (2) is a direct 
sum of irreducible representations, that is 

'''|S0(2)XS0(2) = £h,fel ® ^hM ® • • • ® E/n.fen! 

for some ii, . . . , Z„, A;i, . . . , fc„ e Z. Let u e r be an (S0(2) x SO(2))-eigenvector with eigenvalue e'('i^+'=i''). 
Then the equation 

TR{-0,-ri) = R{9,r])T 

tells us that t{T)v is also an SO(2)-eigenvector, with eigenvalue e~*('i^+'^i''). If T(r)v is a scalar multiple 
of V then li = ki = and the equation T'^ = I tells us that r = triv or 5. Otherwise t|so(2)xSO(2) = 
Eii.fci ® ff-ii -fci and r = (5ij,fei. 

For Z = fc = we have ^o,o — triv © 5. For all other values of I and k the representation Si^k is irreducible. 
Also, for l,k G Z we have that 6i^k is unitarily equivalent to 5-i-k- There are no other equivalences 
between the representations 5i^k- 

The proposition follows. □ 



1.6 Infinitesimal characters 

Let G be a connected reductive group with maximal compact subgroup K, real Lie algebra and 
complexified Lie algebra q. Let tt be a representation of G. Let t^k be the (g, -ftr)-module of iT-finite 
vectors in tt. 

Suppose that tt is an admissible representation of G such that it{Z) acts as a scalar on -kk for all Z in 
the centre ic, of the universal enveloping algebra of q. In particular this condition is satisfied when tt is 
irreducible admissible. Let then [) be a Cartan subalgegra of q and let e [)* be a representative of the 
infinitesimal character of tt. 

Let G = SL4(M) and the subgroups K and P = MAN be as above. Let [) be the diagonal subalgebra 
of g. In this case the Weyl group W{q, fj) acts on () by interchanging elements of the leading diagonal. 
We shall see in the next section that the analytic properties of the zeta functions considered there are 

related to the infinitesimal characters of elements of G. The following proposition gives us information 
about these infinitesimal characters which will be required in the following section. 

Let Pp be the half-sum of the positive roots of the system (g, a), where a is the complexified Lie algebra 
of A, so that 

/ a \ 

pp = 4a. 

—a 

\ -« / 

Let (T be a finite dimensional virtual representation of M , whose Km types are all contained in the set 
{triv,S,Si^k ■■ l,k e {0,2}}. Let be the subset of all C e M such that tr^(/^) = for all Euler- 
Poincare functions fa for cr on M. (Note that the value of tr^(/o-) = depends only on ^ and a and 
not on the choice of Euler-Poincarc fimction Let G^ be the set of all elements of G except for: 

the trivial representation; representations induced from parabolic subgroups other than P = MAN and 
representations induced from ^ e M^- We define an order on the real dual space of a by A > /i if and 
only if (A — /ti) = tpp for some t > 0. 
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Proposition 1.12 Let a be a finite dimensional virtual representation of M and let it G G„. Then the 

infinitesimal character of n satisfies 

Re(u;A„)|a > -^Pp or - pp > Re(wA^)|o 

for every w G W{q, f)). 

Proof: We shall prove the proposition by considering different cases in turn. By Theorem 1.2 we have the 
following cases to consider. The case when tt is a principal series representation induced from P = MAN; 
the case when tt is a complementary series representation induced from P = MAN; the case when tt is 
a limit of complementary series representation and the case when tt is one of the representations Wm for 
m e N. 

First we consider the principal series representations. Let = Indp(^ u) be induced from P, 

where ^ is an irreducible, unitary representation of M = S(SL^(]R) x SL^(IR)) and € a* is imaginary 
(ie. h' e ia^, where aj^ is the real dual space of a). 

Let ^' be an irreducible subspace of ^|sl2(R)xSL2(R)- Then ^' has infinitesimal character A^/ and A^ = A^/. 
We have that C — ^2 where and ^2 are irreducible unitary representations of SL2(K). 

To limit the possibilities for ^1 and ^2 that we need to consider we use the double induction formula (see 
[26], (7.4)). 

Lemma 1.13 (Double induction formula) 

Let M(^A^N(, be a parabolic subgroup of M , so that M(){A()A){N(,N) is a parabolic subgroup of G. If cr^ 
is a unitary representation of M^, and i/« G = (Liec^«)*, v & a* , then there is a canonical equivalence 
of representations 

Ind MAiv (ind m^a^n^ i'^o ® ^o) ®v^= Ind m^(a^a){n^n) i'^o {^o + ^)) ■ 

□ 

Wc may assume that neither ^1 nor ^2 arc induced since then, by the double induction formula, we would 
have the case that tt was induced from a parabolic other than P = MAN, which was excluded. By 
Theorem 1.1, the remaining possibilities for ^1 and ^2 are the trivial representation and the discrete series 
and limit of discrete series representations. 

Let Ajr be the infinitesimal character of tt and A^^ be the infinitesimal character of then A^ = A^^ +A^2- 
Recall that f) is the diagonal subalgebra of q. We lift Aj and to f) by defining Aj to be zero on a and v 
to be zero on the diagonal elements of m, so that A^ = Aj + ([26], Proposition 8.22). The Weyl group 
W = W{q, i)) acts on A^. Let w G W, we will show that either tr^(/(^) = or 

Re(wA^|„) > -^PP 

or 

-pp > Re(wA^la). 

First we take and ^2 to be the trivial representation. This gives us that A5(diag(s, — s, t, —t)) = s + t. 
If i/(diag(a, a, —a, —a)) = aa, a G A, then 

A„(diag(a, b, c, —a — b — c)) 

. / ,a—b b—a a + b a + 6,, a + b 

= A5(diag(^,^,c+^,-c-^)) + a^ (9) 

a — b ( a + &\ a + 6 
= a + c+— (a + 6). 

If we let w = 1 then from above we see that Re(A7r|a) = 0. If we take w to be the transposition 
interchanging h and c we get 

(X 

u'A7r(diag(a, 6, c, —a — b — cj) = a + b + — {a + c) , 
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the real part of which when restricted to a gives 5 pp. All other Weyl group elements are dealt with 
similarly and we see that — ^pp < Rc(A7r|a) < ^Pp for all w G W. 

It remains to consider the case when either or both of ^1 and ^2 are a discrete series representation or 
limit of discrete series representation with parameter rrn . We set = H^q = triv and let = ^+ . or 
= ^TOi' where rrii > 0, and mi and 1712 are not both zero. From (4) we get 

dimpM 

tr$(/a)= E (-ir dim(y5® /\%m®k) ■ (10) 

We shall use Proposition 1.11 to examine the Km types to limit the possibilities for mi, m2 for which 
trC(/a)7^0. 

Lemma 1.14 For mi, 7712 both non-zero we have the following isomorphism of Km -modules: 

n = © s.j,k. 

j>mi ,j = Tnimod2 
k>m.2 ,fc = Tn.2inod2 

If mi =0 we have 

fc>m2,fe=m2mod2 

and for m2 = we have 

n= '^..0. 

_;>mi ,j=mimod2 

Proof: This follows from Lemma 1.8. □ 
Lemma 1.15 We have the following isomorphisms of Km -modules: 



A 

A PM 


= triv 




= ^2,0 ® So,2 


A 2 

A PM 


= S ® 6 ® 62,2 ® S2,-2 


A 3 

A 


= ^2,0 ® (50,2 


. 4 

A 


= triv. 



Proof: Km acts on pM by the adjoint representation and we can compute 

pM — ^2,0 ® ^0,2- 

The other isomorphisms follow straightforwardly from this. □ 

Let V = vi V2 V3 & ® /\* pM Vcr, where the Vi's all lie in a single Km type of their respective 
representation spaces. Lemma 1.15 tells us that V2 is in one of the following Km types: triv, S, 62,0, <5o,2, 
'^2,2, <52,-2 by our assumption on a, the possibilities for the Km type containing V3 are also the same. It 
follows that tr^(/(^) is non-zero only if mi,m2 € {0,2,4}. 

It follows from the exact sequences in the proof of Lemma 1.8 that &~ C ^='='(™^i)''i ^ where the 
index on is + if m is even and — if m is odd. The infinitesimal character of ^±>(™-i)pi is simply 
(m — l)pi (see [26], Proposition 8.22), hence it follows that the infinitesimal characters of ^+ and 
are both equal to (m — l)pi. This gives us in the cases where tr ^(/o-) is non-zero: 

Ag(diag(s, -s,t, -i)) = (mi - l)s-|- (m2 - l)t, mi, m2 e {0,2,4}. (11) 

By computing the action of the different Weyl group elements we can see that in all cases either wA^ \ a > 
— \pp or wA^la < —pp. Since K^^ = K^-\-v and v is imaginary we see that Re(A^|a) = A^|a, so the claim 
follows. 
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The complementary scries representations are dealt with similarly. We recall from Theorem 1.2 that the 
complementary series induced from P are tt = Ind p {itm ®tpp),ioY m and < t < \. We may argue 
as above to find the possibilities for m such that tvitmifa) 7^ 0. In this way we see that there are two 
possibilities for to for which trTt„i{fa) 7^ 0, namely m = 2 and m = 4. In the first case, wA^|a > — jpp, 
for all w G ()). In the second case we have 

A^(diag(a, 6, c, -a - 5 - c)) = (3 + 2i)a + 2tb + 3c. 

If w e f)) is the element which swaps the first and fourth diagonal entries then wA^rlo = — |pp- In 
all other cases we have wA^ 



> 



The limit of complementary series representations are closely related to the family of representations tt™, 
TO e N, so we shall deal with them together. 

For TO, G N, we denote by Tfm the representation of M induced from the representation (g) 
of SL2(M) X SL2(M). For to G N we have the limit of complementary series representation given as 
an irreducible unitary subrepresentation of Im = Indp(7fm, (S) \p)i which we will denote by tt^. The 
representations tt^ are the Langlands quotients of the /„. 

Let A^^ be the infinitesimal character of -Km and A^^-c^ the infinitesimal character of tt^. Clearly A^r^ = 
Ajr^ = Aff^ + |pp, so we need only consider A^^. The value of 



(a 



\ 



is equal to 



a+6+2c 
2 



a+fe-2c 
2 



a+b 
2 



g+b 
2 



a+b 
2 



2 / 



Restricting to we get 



(to — 1)- 



+ (to-1)(c+^— ) + (a + 6) 



= TOa + 6+(TO— l)c. 
(a 

V 



= 2a = -pp. 



■a y 



The Weyl group W acts on A^^. Now for all elements w except one we get wh.Tr^\a > —\pp for all 
TO > 1. The exception is wi which swaps the first and fourth diagonal entries. For this we have 



\ 



\ 



-a J 



-2(to- l)a. 



Thus wiAjr^la > — 5PP if TO = 1,2 and if to > 3 then wiA^^|„ < —pp. 
This completes the proof of the proposition. 



□ 



2 Analysis of the Ruelle Zeta Function 

Let G = SL4(R) and F c G be discrete and cocompact. Let 0k = sl4(IR) and g = sl4(C) be respectively 
the Lie algebra and complexified Lie algebra of G. As in the previous section, all Haar measures will be 
normalised as in [19], §7. Recall that this normalisation depends on the choice of an invariant bilinear 
form b on g. Let b be the following multiple of the trace form on q: 



b{X,Y) = 16tr {XY). 



(12) 
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We choose this normahsation in order to get the first zero of the Ruclle zcta function at the point s = 1 
in Theorem 2.11 below. Let if C G be the maximal compact subgroup S0(4). Let {r C 0r be its Lie 
algebra and let pR C Qm. be the orthogonal space of Sr with respect to the form b. Then b is positive 
definite and Ad(_fi')-invariant on pg and thus defines a G-invariant metric on X = G/K, the symmetric 
space attached to G. 

Let 

/a \ 

a 

^-1 



A= i 



\ 



B 



a > 



S0(2) 



S0(2) 



B is a compact Cartan subgroup of 



M 



{x,y) e Mat2(lR) x Mat2 



det (x),det (y) = ±1 
det (a;)det (y) = 1 



We also define 



N 



h Mat2(K) 
h 



and N 



h 

Mat2( 




h 



and set Km = KCiM. 

Let m denote the complexified Lie algebra of M and let m = ® Pm be its polar decomposition, where 
Im is the complexified Lie algebra of Km — K C] M . Let f) be the Cartan subalgebra of Q consisting of 
all diagonal matrices, and let a, n and n be the complexified Lie algebras of A, N and N respectively. 
Let P denote the parabolic with Langlands decomposition P = MAN and P the opposite parabolic with 
Langlands decomposition P = MAN. Let pp be the half-sum of the positive roots of the system {q, a), 
so that 

/ a \ 

a 



Pp 



V 



= 4a. 



-a ) 



Let 



l -1 



V 



-1 



G aR. 



1 / 



Then it follows that h{Hx) = b{Hi,Hi) = 1 and /5p(Fi) = -i. Let A' = {cxp{tHi) : t > 0} be the 
negative Weyl chamber in A. Let <fp(r) be the set of all conjugacy classes [7] in F such that 7 is conjugate 
in G to an element a^b-y of A~B. 

An element 7 G F is called primitive if for 5 e F and n € N the equation 5" = 7 implies that n = 1. 
Clearly primitivity is invariant under conjugacy, so we may view it as a property of conjugacy classes. 
Let Sp{T) C Spiv) be the subset of primitive classes. 

For [7] G fop(F) we define the length of 7 to be = 6(loga^, loga-y)^/^. Let G-y be the centraliser of 7 in 
G, let F^ = r n G-Y be the centraliser of 7 in F and let Xi(r7) be the first higher Euler characteristic as 
in the previous section. 



2.1 The Selberg trace formula 

Let G be the unitary dual of G. The group G acts on the Hilbert space L^(F\G) by translations from 
the right. Since F\G is compact this representation decomposes discretely: 

L\T\G) = ^Nr{7r)n 



18 



with finite multiplicities iVr(7r), (see [17]). 

Recall that for 7 € F we denote by and the centraliser of 7 in G and T respectively. For a function 
/ on G, denote by 0^{f) the orbital integral 



The Selberg trace formula (see [40], Theorem 2.1) says that for suitable functions / on G the following 
identity holds: 



where the sum on the right is over all conjugacy classes in T. The set of suitable functions includes, but 
is not limited to, all dimG + 1 times continuously differentiable functions of compact support on G. In 
fact, we shall need to extend the set of test functions for which the trace formula is valid. 

Lemma 2.1 Let d G N, d > 16, that is d= 2d' for some d' > dimG/2. Let f be a d-times differentiable 
function on G such that Df G L^{G) for all left invariant differentiable operators D on G with complex 
coefficients and of degree < d. Then the trace formula is valid for f. 

Proof: This is Lemma 1.3 of [10] in the case G = SL4(M). □ 

For g G G and V any complex vector space on which g acts linearly let E{g\V) C M+ be defined by 

E{g\V) = : /i is an eigenvalue of g on V}. 

Let Aniin {g\V) = mm{E{g\V)) and Xn.,A9\V) = max{E{g\V)). 
For am e AM define 



where we are considering the adjoint action of G on n and g resepectively. Define the set 

(AM)" = {am G AM : X{am) < 1}. 

An element of G is said to be elliptic if it is conjugate to an element of a compact torus. Let M^u denote 
the set of elliptic elements in M. 

Lemma 2.2 The set (AM)"" has the following properties: 

(1) A-Meii C {AM)-; 

(2) am e {AM)"' a e A"; 

(3) am,a'm' G {AM)'^,g G G with a'm' = gamg~^ ^ a = a' ,g G AM. 

Proof: See [8], Lemma 2.4. □ 

For the construction of our test function we shall need, for given s e C and j G N, a smooth, conjugation 
invariant, j-times continuously differentiable function on AM, with support in {AM)"' . Further, wo 
require that at each point ab G A~ B the function takes the value ^^^^e"^'". In [8] a function gi is 
constructed with these properties, with the one difference that there the positive Weyl chamber is 
used instead of the negative Weyl chamber A~ . With only very minor modification the construction in 
[8] will yield a function with our required properties, which we shall also call g^ . 

Let rj : N ^ M. he a smooth, non-nogative function of compact support, which is invariant under 
conjugation by elements of Km and satisfies 



Let / : M — > C be a smooth, compactly supported function, invariant under conjugation by Km- Suppose 
further that the orbital integrals of / on M satisfy 





(13) 





M/M, 



m 
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whenever m is not conjugate to an element of B, where Mm denotes the centraUser of m in M. The 
group AM acts on n according to the adjoint representation. 

Given these data we define $ = ^f,j,s : G ^ C by 
for fc e -fC, n e N, am e AM. 

To see that $ is well defined we recall that, by the decomposition G = KNAM, every g G G which is 
conjugate to an element of (AM)"' can be written in the form knam,{kn)~^ . By the properties of [AM)"^ 
we see that two of these representations can only differ by an element of Km- The components of the 
function $ are all invariant under conjugation by Km, and we note that det (1 — {am)~^\n) ^ for all 
am e {AM)'" , so we can see that $ is well-defined. 

Proposition 2.3 The function $ is {j — 14) -times continuously differentiable. For j and Re(s) large 
enough it goes into the trace formula and we have: 

^NrMt.m- S vol(r,\G,X(/) (15) 

TTSG [7]e<rp(r) v 7 7/ i ; 

Proof: Noting that 2dimn + dimJ = 14 we see that this follows from Proposition 2.5 of [8]. This propo- 
sition was proved in the case that the function / is an Euler-Poincare function for some finite dimensional 

representation of M. However the only properties of Euler-Poincarc functions used in the proof are those 
given above for /, namely that it is smooth, of compact support, invariant under conjugation by K, and 
the orbital integrals satisiy the given condition. □ 



2.2 The Selberg zeta function 

An element g G G is said to be weakly neat if the adjoint Ad(5f) has no non-trivial roots of unity as 

eigenvalues. A subgroup of G is said to be weakly neat if every element of H is weakly neat. Let 
[7] € <?p(r) so that 7 is conjugate in G to a^b^ G A~B. We want to know which roots of unity can occur 
as eigenvalues of Ad(7). These are the same as the roots of unity which occur as eigenvalues of Ad{ajbj). 
If 

R{9) 

where 

_ f cos 9 —sm9 
^ ' y sm9 cos 9 

then the eigenvalues of Ad(a-y6-y) are e^^'^ and e^^'*^. Define the sets 

R0 = {nGn:n9 G ttZ} and R^ = {nGn:n^G ttZ} 

and 

Rj = {mini?0, mini?^}. 

Then R~^ contains either 0, 1 or 2 elements. We can see that 7 weakly neat is equivalent to R^ = 0. For 
7 € (#p(r), where 7 = 7o for 70 primitive, the value of xi(r^) depends on whether or not n G R-^fa- 

For I C Rj with 1^0 define nj to be the least common multiple of the elements of / and set = 1. 
Further, define 

X/(7) = ^^E(-l)'"^i(r7''^)- 

Let z e C\ {0} and q G Q. We define z'' to be equal to e''°s^, where we take the branch of the logarithm 
with imaginary part in the interval (— tt, tt]. 

For any finite-dimensional virtual representation a of M we define, for Re(s) large, the generalised Selberg 
zeta function 

ZpAs) = n n n (1 - e-^"''-7"' \V, ® ^"(n))'^'^^^ , (16) 
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where S'"(n) denotes the n symmetric power of the space n and 7 acts on 5"(n) via a{b^) 
Ad" {a.ybj) . In the case that F is weakly neat this simplifies to 



XiiT-,) 



(see [8]). 

Let TT £ G. The Lie algebra n acts on ttk and we denote by H* (n, ttk) (resp. H, (n, ttk )) the corresponding 
Lie algebra cohomology (resp. homology) (see [3], [4]). We have the following isomorphism of j4M- modules 
(see [20], p57): 

Hp{n, wk) = H^-^n, wk) ^ /\ n. (17) 

For A e 0* let 

mx{n) = J2 (-l)^+''dim (if''(n,7r^)^ ^ f\%M ^ V^^" , (18) 

p,q>0 

where for an a-module V and A G a* , denotes the generalised A-eigenspace 

{t; e V| 3n e N such that (a - X{a))'^v = Va e a} 
and the superscript Km denotes the subspace of Km invariants. 

We say that an admissible representation tt of a linear connected reductive group G' has a global character 
6 = 9^ if for all smooth, compactly supported functions / on G' the operator n{f ) is of trace class and 
9^ is a locally function on G' satisfying 



tr7r(/)= / Q'^'{g)f{g)dg 
Jg' 



for each such /. By [26], Theorem 10.2, every irreducible admissible, and in particular every irreducible 
unitary, representation of G' has a global character. 

Theorem 2.4 Let a be a finite dimensional virtual representation of M. The function Zp^a-{s) extends 
to a meromorphic function on the whole of C For A e a*, the vanishing order of Zp^„{s) at the point 
s = A(iJi) is 

J2 Nr{7T)mx{7r). (19) 
Further, all the poles and zeros of Zp^„{s) lie m K U (5 + iR). 

Proof: The analogue of this theorem in the case that G has trivial centre and F is weakly neat is proved 

in [8], Theorem 2.1. If G has trivial centre then F weakly neat implies F torsion free, since non weakly 
neat torsion elements must be central. With a few modfications the proof of [8], Theorem 2.1 becomes 
valid in our case also. In fact the assumption that F is weakly neat is used in two places. We sketch the 
proof here, pointing out the necessary modifications for it to be valid in our case. 

The theorem is proved by setting / = f^ in the test function <& = = '^a.j.s defined in (14), where fa 
is an Euler-Poincare function for a on M. We then get that for j G N and Re(s) sufficiently large, the 
right hand side of (15) is equal to 

E vol(r,\GJ<(/,)^^^^. (20) 

In [8] the following equation from [7] , proved under the assumption that F is torsion free, is used: 

C^yo\{T^\G^) 

XUIt)- -1 ' 1^1) 

'70 

where G-y is an explicit constant depending only on 7, and 70 denotes the primitive geodesic underlying 
7. We have shown in Lemma 1.9 that (21) holds for all 7 S SpiV) in our case also. From [8], Proposition 
1.4 we take the equation 

0^{fa)=C^Ua{b^), 
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which, together with (21) gives us 

0,^(A)vol(r^\G^) = /^„tra(6^)xi(r^). 
Substituting this into (20) we get 

^ ^otra(6,)xi(r,) ^ " 

We claim that this is equal to 

i+2 



/a \ 



Indeed 

oo 

e 

m 



logZp^s) = - E E E X/(7)^^tra(67) E tr {ia,b,r\S^{n)) . 

[7]e^^(r)m=l/cfi7 n>0 

We note that 

E tr {{a^b^)'^\S"{n)) = det (1 - (a^6^)"|n)"^ 

n>0 

= det {1 - {ajb^)-"'\n)~'^ 

and the claimed equality follows. 

In [8], since T is weakly neat it follows that G^^ = for all 7 G F and n E N. In [14] it is shown that 
= Tj\Gj/Kj and so it follows that for all n G N we have Xj^^ = Xj and hence Xi(r7") = Xi(r7)- 
Thus the above equality involving the logarithmic derivative of Zp^„{s) is derived with the simpler Euler 
product expansion for Zp^„{s) given above. 

In our case we may have an element [7] € <^'p(r) with a non-trivial root of unity as an eigenvalue. For 
such a 7 we have Xi(r7") 7^ Xi(r7) for n & R^. For this reason we have had to introduce the more 
complicated Eulcr product expansion for Zp^„{s) so that the above equality involving the logarithmic 

derivative of Zp,^{s) still holds. 

On page 909 of [8] it is shown that 

tr7r($,)=/ Um)@^.\,^,^){ma)dmgi{a)da. (22) 

JMA- 

Using the property (4) of f„ we get 



tr7r($s) = ii {a (H*{n,T^K)® /\ Vm®V^^ 

POO 

= / E^^w^^^^'"''"'^**'^''^* 



li+^e-'^-^ da 



Aea* 

= i-^y*' ^,'"^Mj^y (23) 



Proposition 2.3 then gives us that 



(I;) iogZp.W = E*w(|;) E^-MjdcSo' 



(24) 



from which it follows that the vanishing-order of Zp,j{s) at s = A(ifi) is 

E^r(7r)mA(7r). 

Two further comments are in order. First, in [8] the positive Weyl chamber = {exp(tffi) : t < 0} 
in A is considered, where we have instead considered the negative chamber A~ . For this reason we have 
interchanged the positions of the Lie algebras n and n from the way they are used in [8]. This is easily 
seen to give a precisely equivalent result. 

Secondly, the results of [8] are stated for a finite dimensional representation ct of M. By linearity the 
results extend in a straightforward way to the case where cr is a virtual representation, which we use 
here. □ 
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2.3 A functional equation for Zp^s) 

Proposition 2.5 For A e a* let ||A|| be the norm given by the form b in (12). There are mi e N and 
C > such that for every n & G and every A e a* we have 



4 

^(-l)«dim(i/«(n,7r^)^) 

q=0 



< C{1 + 

Further, let S denote the setof all pairs {n, X) G G x a* such that 



Then there is m2 G N such that 



^(-l)«dini(//''(n,7rK)^) ^0. 

g=0 



E <oo- 

(7r,A)eS 



Proof: These results follow from [11], Proposition 2.4 and Lemma 2.7. □ 

An entire function / is said to be of finite order if there is a positive constant a and an r > such that 
\f{z)\ < exp {\z\'^) for \z\ > r. If / is of finite order then the order of f is defined to be the infimum of 
such a's. 

It is well known that the classical Selbcrg zeta function is an entire function of order two (see [37]). Our 

next lemma gives an analogous result for the generalised Selbcrg zeta function we are considering here. 

Let cr be a finite dimensional virtual representation of M . Theorem 2.4 tells us that Zp^^^^s) is meromor- 
phic and hence it may be written as the quotient of two entire functions: 



Z2{sy 



where the zeros of Zi{s) correspond to the zeros of Zp^a{S) and the zeros of ^2(5) correspond to the 
poles of Zp^a[s). The orders of the zeros of Zi{s) (resp. -^2(5)) equal the orders of the corresponding 
zeros (resp. poles) of Zp^^ris). The functions Zi{s) and Z2{s) are not uniquely determined, but clearly 
their respective sets of zeros, together with the orders of the zeros, are. For i = 1, 2 let Ri denote the set 
of zeros of Zi (s) counted with multiplicity. 

Lemma 2.6 There exist Zi{s), for i = 1,2, with the above properties, which are, in addition, of finite 
order. 

Proof: Let Zi{s), Z2{s) be as above. We shall show that we may take them to be of finite order. 

For TT e G let A(7r) be the set of all Ago* with A 7^ and m\{Tr) ^ 0. Let G{T) be the set of tt e G such 
that Nr{n) ^ and let S denote the set of all pairs (tt. A) such that tt € G(r) and A G A(7r). For A G a* 
let ]|A]| be the norm given by the form b in (12). 

The expression (19) tells us that s 7^ is a zero or pole of Zp{s) if and only if s = X{Hi) for some A G 0*, 
for which there exists tt G G(r) such that (tt. A) G S. 

Since the function Zp^„{s) is meromorphic and non-zero, it follows that there exists c > such that 

lA(ifi)|>c(l + l|A||) (25) 

for all A such that (tt. A) G 5 for some tt. 

By the definition of m\{TT) we deduce immediately from Proposition 2.5 that there exist mi G N and 
G > such that for every tt G G and every A G o* we have 

|m,(7r)|<G(l + l|Al|)™\ (26) 

and that there exists m2 G N such that 

(7r,A)eS ^ 
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Let k = mi + m2- Then, for (tt, A) e S, by (25) and (26) we have, 



mx{Tr) 



< 



|mA(7r)| 



(1 + 



< £. 



1 



(l + i|A||)" 



It then follows from (19) and (27) that, for i = 1,2, 



Sr J-< \^ Nr{TT)\mx{7T)\ 

pGi?ix{0} (7r,A)6S ' ^ ^'^ 



(28) 



Wc say that ^i(s) and Zi(^s) arc oi finite rank. 

By the Weierstrass Factorisation Theorem ([6], VII. 5. 14) and (28), there exist entire functions gi{s), g2{s) 
such that 



Zi{s) 



n 



s s" 



1 - t) t^xp (- + — ••• + — 



s 

kp'^ 



(29) 



where rii is the order of the zero of Zi{s) at s = 0. 
From (24) we recall the equation 



E 



-A(ifi)' 



(30) 



7reG(r) ^ ' AeA(7r)U{0} 

where j e N is sufficiently large. Let J = max(j, k). It follows from (29) and (30) that 

(-l)^-i(J-l)! ^ (-l)^-i(j-l)! 



i9[is)-g'2is))+Y^ 
peRi 



{s-pY E^ {s-pY 

{-iY-\j-i)\ 



is equal to 



E E Nr{Ti)mx{Ti) 

7reG(r) AeA(7r)U{0} 



{s-X{Hi)Y • 



Remembering that the zeros of Zi{s) are included with multiplicity in Ri{s) and bearing in mind the 
expression (19) for the vanishing order of Zp^a{s), we see that this implies that 

\g[{s)-g'2{s)) = 0. 

It follows that gi{s) — 52(5) is a polynomial of degree at most J. Without loss of generality we may take 
172(5) to be zero, so that .gi(.s) is itself a polynomial of degree at most J. Finally, Theorem XI. 2. 6 of [6] 
tells us that since Zi{s) and Z2{s) are of finite rank and gi{s) and 52(5) are both polynomials of degree 
at most J, it follows that Zi{s) and Z2{s) are both of order at most J. □ 



Before we give the next lemma we make a couple of definitions. Let G" be a linear connected reductive 
group with maximal compact subgroup K' and let ^ be a representation of G'. Let g' be the complexified 

Lie algebra of G' , let [)' C g' be the diagonal subalgcbra and let p' G (t)')* be the half sum of the positive 
roots of the system {g',i)'). We say that ^ is tempered if for all /sT'-finite vectors u,v £ ^k' there exist 
constants Cu,v such that for all g G G' 

m9)u,v}\<cu,v I e-P'^^^o'^'^Uk, 
Jk' 

where H{g~^k) denotes the logarithm of the split part of g~^k under the Iwasawa decomposition and 

(•, •) is the inner product on V^. An admissible representation of G' is called standard if it is induced 
from an irreducible tempered representation of M' C G', where P' = M'A'N' is a parabolic subgroup of 
G". 

The Weyl group W{G, A) has two elements, let w be the nontrivial element therein. Then w acts on M by 
conjugation and for a representation cr of M we can define the representation by "'cr(m) = (j{wmw~^). 
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Lemma 2.7 Suppose that a = as Km -modules. Then the functions Zp^„{s) and Zp^„{l — s) have the 
same poles and zeros with multiplicity. 

Proof: By Theorem 2.4 Ago*, the vanishing order of Zp^„{s) at the point s = X{Hi) is equal to 
J2TreG -^r(7r)mA(7r). We shah show that 

mx{Tr) = TO-A^2pp(7r) (31) 

for all TT E G. Since —2pp{Hi) = 1 the lemma will follow. 
Let TT € G. Recah from (18) that 

mx{7r) = Yl (-ir+'dim (F9(n,7r^)^ ^ f\%M ^ V^Y'' . 

P,<1>0 

The global character of tt on G can be written as a linear combination with integer coefScients of 
characters of standard representations ([26], Chapter X, §10.2). We are interested in the values taken 
by 8^ on MA~. By [26], Proposition 10.19, the only characters which are non-zero on MA are those 
characters of representations induced from P = MAN. So there exist n G N and for all 1 < i < n integers 
Cj, tempered representations of M and Ui e a* such that 



e^ = ^c,e^„ (32) 



i=l 



where tt* = Ind p(^i (8) Vi). From [20], Theorem 3.6 and (17) it follows that for all regular ma G MA we 
have 

det (a\ f\^n) 

Q-(-)-^"'K--)(-) det(l-n^a|nr 
and the same holds for ah tt*. Together with (32) this imphes that 

n 

i=l 

for all regular ma G MA~ . Substituting this into (22) and proceeding as in (23) we see that it suffices 
to show (31) for the representations tt*. 

The Weyl group element w acts on the group MA by conjugation, which has the effect of swapping the 
two components. For G a* we therefore have wv = —v. Recall that for a representation ^ of M wc 
let ™^ denote the representation "^(m) = $,{wmw~^). By [26], Theorem 8.38, the representations tt* and 
""tt* = Ind pC"^^ ® {—^i)) are equivalent. Prom the definition of the induced group action we can see that 

F9(n,7ri^)^ = i?«(n,™7rif)-^-2''^. 

Thus we see that 

dim (Hi{n, Trjf )^ ® /\'' Pm O 

is equal to 

dim (Hi{n,^'K)^)-^-^P^ (g: ® Vc 



Km 



Km 



Using the notation of Proposition 1.11 we note that the ifjvf-types satisfy ^trivKM = tfi^xM^ ^detxM = 
detKM aiid = ^m,i — ^-m,-i- Lemma 1.15 and the isomorphism of /^M-modules 

n = '^2,2 ® ^2,-2 

tell us that the /^M-modules /\^ pM and n are invariant under the action of w and we have assumed that 
'^a = a as ii'M-niodules. Since w"^ is the identity element of W{G, A) we can conclude that 

(, . n \ Km 

7J«(n,7rl^)^g /\ Pm<8K.J 

is equal to 

dim {^H^n^n'j.y^-^P" (S> /\%M (S)V^y'' . 
Hence m\{TT^) = m_A-2/)p (tt*) and the lemma follows. □ 
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Theorem 2.8 Suppose that a ^ as Km -modules. Then there exists a polynomial G{s) such that 
Zp,a{s) satisfies the functional equation 

ZpAs) = e^^'^Zp^^il - s). 

Proof: Let Ri, Ui be as above for i = 1,2. We saw in the proof of Lemma 2.6 that there exist entire 
functions Zi{s) and Z2{s) of finite order such that 

and polynomials gi{s) and 52(5) such that 



pe-Ri\{0} 



Lemma 2.7 tells us that Zpcr(s) and Zp^„{l — s) have the same poles and zeros. Hence we can in the 
same way conclude that there exist entire functions Wi{s) and W2{s) of finite order such that 

and polynomials /ii(s) and /i2(s) such that 

pe-R,\{o} ^ \f r f / 

Setting 

G{s) = gi{s) + h2{s) - 52(5) - /ii(s) 
we can see that the claimed functional equation does indeed hold. □ 

2.4 The Ruelle zeta function 

For any finite-dimensional virtual representation cr of M we define, for Re (s) large, the generalised Ruelle 
zeta function 

RrAs) = n n det (1 - e-'"'^-r' IV,)""^'^ . 
[7]e<f^(r) ICR., 

We have the following theorem giving a relationship between the generalised Selberg zeta function and 
the generalised Ruelle zeta function. 

Theorem 2.9 Let a be a finite dimensional virtual representation of M. The function Rr,a-{s) extends 
to a meromorphic function on C. More precisely 



1 j 



,(-1)' 



Proof: Let 7 G S'p{T) let /i(7) e N be the least such that 7 = 7o for some 70 € ^p(r). We compute 
at first 

logRrAs) = Xi{l)trlog{l-e-'^''-r'\Va) 
= E E X/(7)-7-^tra(6-). 
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Similarly, for Tq= /\^ n (g) ct we have that log -^p.t, (s) equals 



OO OO ^ 

E E X^WE E — e-^™-'nr {i^-'\Vr^®S%n)) 

b]eSl(T) ICR-, n=0 m=l 

tr(6^^| A'n) 



u(7) ^ ^ ^det (1- (a^6^)-"'|n) 



Since n is an M-module defined over the reals we conclude that the trace tr ((a^6^)~^| n) is a real 
number. Therefore it equals its complex conjugate, which is tr (a~^b-y\ /\* n). Summing over q we get 

4 

logi?r,a(s) = ^(-l)«logZp,,, + I) • 

g=0 

The theorem follows. □ 

We shall be interested in the zeta function Rp^„{s) in the case where a = triv = 1 is the trivial represen- 
tation of M and in the case where a is the following virtual representation of M. Define 

a+ = 15/\ m+6/\ tn+/\ m 

and 

CT" = 10 /\' m + 3 /\' 

and let cr = cr+ — where M acts on /\" m according to Ad". The reason for this choice of a is that 
it allows us to separate the contribution of the non-regular elements of S'p{T) from the regular ones, as 
made clear in the following lemma. 



3 

m 



Lemma 2.10 Let 7 e S'piT) with 

bj - 

where 

m 

Then 



R{e) 

m J ' 

cos 9 — sin 6* 
sin 9 cos 9 



tia{bj) = det (1 - 6^ : m/b) = 4(1 - cos26')(l - cos 2^), 

where b is the complexified Lie algebra of the group B. In particular we have tva{b-y) > for all 
7 e ^p(r), and tr cr(6^) = if and only i/7 is non-regular. 

Proof: As B-modules, we have the isomorphism m = b ® (iri/b). The group B is abelian and so acts 
trivially on the 2-dimensional Lie algebra b. Hence, for < n < 6, we have the following isomorphism of 
i?-modules 

A"-- e C)A'-/b. (33) 

p-\-q=n ^ ^ 

We set ao = 1, ai = —3, a2 = 6, as = —10, a4 = 15 and note that, for = 0, 4, these satisfy 

E ( ' ) = (34) 

m=0 ^ ^ 

The B-module isomorphism 

4 

A * X — ^ A ^ 

/\ (m/b) = 2^ a4-„ /\ m = CT 

n=0 



27 



then follows from (33) and (34). We can then see that 

tva{b-y) = tr (^bj : ^ tn/b^ = det (1 — 6-y : m/b). 

The adjoint action of B on m/b can easily be computed to give the claimed value. Finally we note that 
tr (T{b^) = if and only if 9,(j) G Ztt which is equivalent to 7 being non-regular. □ 

The main result of this section is the following theorem. 

Theorem 2.11 The function Rr,i{s) has a double zero at s = 1. The function Rr,a{s) has a zero of 
order eight at s = 1. Apart from that, for cr G {1, ct}, all poles and zeros of Rr^ais) are contained in the 
union of the interval [—1, |] with the five vertical lines given by j + iM. for k = —2, —1, 0, 1, 2. 

The theorem will follow from the following proposition, which we prove in the remainder of the section. 

Proposition 2.12 The function Zp^i{s) has a double zero at s = 1. The function Zp^^(s) has a zero 
of order eight at s = 1. Apart from that, for a G all poles and zeros of Zp^a{s) lie in the half- 

plane {Rc(s) < |}. Further, for Tg being the representation of M on /\'' n {q E {1,...,4}) obtained 
from the adjoint representation and a G {Tq,Tq (g) a}, all poles and zeros of Zp^o-(s) He in the half-plane 
{Re(s) < 1}. 

Let us assume for a moment that the proposition has been proved. The representations a of M considered 
in the proposition all satisfy the isomorphism of ii'M-modules a = ^a, where w is the non-trivial element 
of the Weyl group W{G,A). Hence we can apply the functional equation given in Theorem 2.8 to see 
that the poles and zeros of the functions Zp^cr{s) all lie in the region < Re(s) < 1. We can then apply 
Theorem 2.4 to see that the poles and zeros in fact lie in [0, 1] U (5 -|- zR). Finally, an application of 
Theorem 2.9 completes the proof of Theorem 2.11. 

The proof of the proposition will take up the rest of the section. We see from (19) that a representation 
TT e G makes a contribution to the vanishing order of Zp^o-(s) only if mxiTr) 7^ for some A G a*. From 
(23) it follows that, if tT'K{^s,a) = for some it G G and finite dimensional representation a of M, then 
m\{iT) = for all A <E a*. Thus, if we can show that tr 7r($s.cr) = for some tt and cr, then we will know 
that TT makes no contribution to the vanishing order of Zp^„. 

Since pp{Hx) = — i, we shall be able to prove Proposition 2.12 with the following steps. Firstly, irre- 
spective of the choice of a, all eigenvalues A of on H*{n,'KK) satisfy 

Re (A) > -2/9p, (35) 

with equality only if tt = triv. Secondly, in the case a G if tt is the trivial representation on G, 

then ff^(n, 'Kk)~'^''^ 7^ 0. If cr = 1 then this gives a double zero at the point s = 1 and if ct = a this gives 
a zero of order eight at s = 1. Apart from this, for all tt S G and all A S a*, we have either tr 7r($) = 0, 
or that H'{n,iTK)^ 7^ implies 

Re(A) > -'^pp. (36) 

The first step will be proven using the Hochschild-Serre spectral sequence in the following two sections. 
Then the second step will be proven using a result of Hecht and Schmid which relates the action of a on 
H*{n,'KK) to the infinitesimal character of tt. 

2.5 The Hochschild-Serre spectral sequence 

We shall need the following proposition. 

Proposition 2.13 Let P' = M'A'N' be a parabolic subgroup of G and let a' ,n' be the complexified Lie 
algebras of A',N' respectively. Define a partial order >a' on the dual space a'* of a' by p, > u if and only 
if II — V is a non-zero linear combination with positive integral coefficients of positive roots of (g, a). 

Let A G 0'* and < p < dimn be such that if^(n, ttk)^ 7^ 0. Then there exists p, G a'* with A >„' p. such 
that iJ'i"""(n,7rx)'' 7^ 0. 
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Proof: This follows from [20], Proposition 2.32 and the isomorphism (17). □ 

Let Mo C G be the subgroup of diagonal matrices with each diagonal entry equal to ±1, let C G 
be the subgroup of diagonal matrices with positive entries and let A^o C G be the subgroup of upper 
triangular matrices with ones on the diagonal. Then Pq = MqAqNq is the minimal parabolic subgroup 
of G. Let Oo and no be the complexified Lie algebras of Aq and A^o respectively, and let Oq be the dual 
of Oo- Let Po e Oq be defined as follows: 



Po 



fa \ 



3a + 26 + c. 



Then po is the half-sum of the positive roots of the system (g, Oq). Let Oo,k be the real Lie algebra of Ao, 
which we may consider as a subalgcbra of ao- Lot a'^^ be the set of all X E ao.R such that a{X) < for 

all positive roots a of the system (g, ao)- Let be the set of all v G a^^ such that I'iX) < for all 

X e Oq K- Then for all f e Oq'^ we have v = J2a ^aCiy where the sum is over the positive roots of (g, ao) 
and every > 0. 

For TT e G a matrix- coefficient of tt is any function G — > C of the form 

fu,v{9) = {■K{g)u,v) 

for some u, w € tt. 

Lemma 2.14 Let tt € G and let A G a^ &e such that H*{x\o,'kk)^ 0. Then A G — 2po + 
except in the case w = triv, when H^{no,7rK)~^^° 7^ and other than this H*{no,triv)^ ^ implies 
AG -2po + ao;i. 

Proof: According to [20], Theorem 4.16, there exists a countable set S'{n) C and a collection of 
polynomial functions p'^ ^ indexed hy u,v G ttk and v G ^(i") such that if for u,v G ttk, is a matrix 
coefficient of tt then: 

/„,„(expX)= PuAX)e^''^'"^^^''\ (37) 

for all X G cI(7r- We assume that S'iir) is minimal and hence that each polynomial pj^ ^ is non-zero. 

If TT = triv then every matrix coefficient of tt is a constant function so in (37) we have # (tt) = {—po} and 
p"^" = Cu,v, where Cu,v is a constant for all u,v G w. 

If TT G G \ {triv} then tt is infinite dimensional so by [25], Theorem 5.1 all the matrix coefficients of tt 
vanish at infinity. Hence, for all v G S'{Tr) and for all X G agg, we have {u + po){X) < 0. In other words, 

for all V G we have v G —po + Aq r- 

We have assumed that H* {wotTTk)^ 7^ 0. Let be the partial order defined on aQ by p v if and 
only if /X — i/ is a linear combination with positive integral coefficients of positive roots of (0,00). By 
Proposition 2.13 there exists /U G Og such that if^(no, tt/c )** 7^ and A >„(, p. 

Let A = {j^ + po G a^ : i?*'(no, t^kY 7^ 0} and let A™'" be the set of elements of A which are minimal with 
respect to >ao- Let (^""'"(Tr) be the set of G S{t^^ which are minimal with respect to >aa- Theorem 4.25 
of [20] says that A™ = (?(7r)'"'". 

If TT = triv then A™'" = {— po} and the claims of the proposition follow from the definition of A and [20], 
Proposition 2.32 as quoted above. 

If TT G G \ {triv} then A""*" = (?(7r)'"'" implies that there exists v G S{'p:) such that A >„o - po- We 
saw above that v G —po + ^° claim follows. □ 

Proposition 2.15 Let tt G G and let A G 0* fee such that H'{n,-KK)^ 7^ 0. Then Re(A) > —Ipp, with 
equality only if n = triv. 

Proof: Let riAf = no n m and aM = ao n m. Then no = nAf ® n and ao = a ® Om- In [23, 22] a filtered 
complex is constructed so that the spectral sequence derived from it has 
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and 

where i?P+'(no, ttk) is appropriately filtered. This is the so-called Hochs child- Serre spectral sequence. 
By Proposition 2.13 it suffices to prove the proposition in the case 

ir4(n,7rx)V0. 

In this case it then follows that 

H\nM,H\n, tt^^ )^) ^ 

and so there exists Am G O-m such that 
Since A acts trivially on xim, this equals 

where we consider A + Am as an element of ttg = a* ® • For all r we have that E^''' ^ only if < p < 2 
and < g < 4, so it follows that £^2'^ = E'^. Since the action of Aq commutes with the differentials of 
the spectral sequence it follows that 

and hence that 

The proposition then follows from Lemma 2.14 by projection of A + Am onto the 0* component. □ 

2.6 Contribution of the trivial representation 

For a ii'M-module r] let 2r] denote the module r}®r}. We shall need the following: 
Lemma 2.16 We have the following isomorphisms of Km -modules: 

/\ pM = triv 

f\ pM = ^2,0 e 5o,2 
A 2 

/\ pM = 25 52,2 52,-2 
A ^ 

l\ pM = ^2,0 ® 5o,2 
. 4 

/\ pM = triv. 

and 







triv 






26 ® 52,0 ® '^0,2 


A 2 

A ^ 




triv © 25 ® 2(52,0 ® <5o,2) ® 52,2 © 52,-2 






2triv © 2triv © 2(52,o ® '^0,2 ® 52,2 ® 52,-2) 






triv © 25 © 2(52,0 ® ^0,2) ® 52,2 ® 52,-2 



Proof: The isomorphisms for pM were given in Lemma 1.15. For m note that Km acts on m by the 
adjoint representation and we can compute 

m = 25ffi52,offi5o,2- 

The other isomorphisms follow straightforwardly from this. □ 
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Proposition 2.17 Let tt be the trivial representation on G. 

For a the trivial representation on M, the representation tt contributes a double zero of Zpjj{s) at the 
point s = 1. For a — a the representation tt contributes a zero of Zp^„{s) of order eight at the point 
s = 1. In hath cases, all other poles and zeros contributed by tt are in {Re(s) < §}• 

Proof: The space Ho{n,'KK) = t^k/^t^k is one dimensional witii trivial a-action. The action of a on n 
is given by 5/Op, so the isomorphism (17) tells us that a acts on the one dimensional space i?*(n, ttx) 
according to —2pp. Proposition 2.13 tells us that for g = 0. 1,2,3 and A G a*, H'^{n,'KK)^ 7^ implies 
A > — |pp- Since —2pp{Hi) = 1 this gives a pole or zero at s = 1 and evaluation of the relevant 
characters at Hi also gives the other poles and zeros contributed by tt in {Re(s) < |}. 

It remains to compute the vanishing order of Zp^„{s) for a S {1)5'} at the point s = 1. Since 
dimi?'^(n, TTif )~^''^ = 1, we get from (19) the following expression for the vanishing order: 

^^r(7r) Y.^-\f dim ( A" • 

p>0 

For cr = 1 this is equal to 

iVr(7r)^(-l)^'dim(/\%M)''" 

p>0 

and for cr = a to 

^r(^) (- 1)^+' dim (/\%M0ag A' m)"^", 

P>9>0 

where 0,0 = 15, a\ = 10, 02 = 6, 03 = 3, 04 = 1 and Oq = for q > 5. The only functions on L'^(T\G) 
invariant under the action of G are the constant functions, hence iVr(7r) = 1. We can then use Lemma 
2.16 to see that the above expressions take the claimed values. □ 

2.7 Contribution of the other representations 

The following proposition gives a relationship between the action of on -ff*(n, ttk) and the infinitesimal 
character of tt. 

Proposition 2.18 Let tt G G. 

Suppose H*{n,nK)^ ^ for some A e 0*. Then A = wA.^\a — pp, where w G W{g,l)) and A„ is a 
representative of the infinitesimal character of n. Moreover, for p gZ we have 

weW{g,t)) 

Proof: This follows from [20], Corollary 3.32 and the isomorphism (17). □ 

In light of this proposition, in order to complete the proof of Proposition 2.12, and hence of Theorem 
2.11, it will be sufficient to show that for all tt G G \ {triv} and for a G {1, a} either trTr{^a) = or the 
infinitesimal character A^r of tt satisfies 

Re(«;A^)|„ > -^Pp or - pp > Re(wA^)|a 

for all w e W{q,1)). 

We first consider the elements of G induced from parabolic subgroups other that P = MAN. 
We say that two paraboUcs P' = M'A'N' and P" = M"A"N", where 

A', A" C ^0 = {diag(a,6,c, {abc)~'^)\a,h,c> 0} 

are associate if there exists a member w of the Weyl group W{q,\)) such that w~^MAw = M'A'. 
Representations of G induced from associate parabolics are equivalent. 
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Up to association G has four parabolic subgroups: P defined above; the minimal parabolic Pq = MqAqNq 
where Mq = {±1} and A^o is the group of real, upper triangular matrices with ones on the diag- 
onal; the parabolic P' with Langlands decomposition P' = M'A'N' where M' ^ SLf (M) x {±1}, 
A' = {dia.g{a, a,b, a^^b^^)\a,b > O} and N' is the group of real, upper triangular matrices with ones 
on the diagonal and otherwise with zeros in the second column; and the parabolic P" with Langlands 
decomposition P" = M"A"N" where M" ^ SLf (R), A" = {diag{a, a, a, a-^)\a > O} and N" is the 
group of real, upper triangular matrices with ones on the diagonal whose only non-zero entries above the 
diagonal are in the rightmost column. 

Proposition 2.19 Let tt be a principal series or complementary series representation induced from the 
parabolic P = MAN, where P = Pq, P' or P" and let a be a finite dimensional representation on M . 
Then tT7r{^cr) =0 so w contributes no zeros or poles to Zp^„{s). 

Proof: Let &^ be the global character of the irreducible unitary representation n, so that 

tr7r($o-)= / ^a{x)QTv{x) dx. 
Jg 

The Weyl integration formula can be applied (see [8] , p908) to give us 

tr7r($^) = / / fa {mlm~^) dm e^{al)d{al) dl, 

^ \ W[L)\ J A+L J M/L 

where the sum is over conjugacy classes of Cartan subgroups L of M, and we denote by W{L) = W{L, M) 
the Weyl group of L in M, by fcr an Eulcr-Poincarc function for a and d{al) is an explicitly given function 
on A'^L. Proposition 1.4 of [8] tells us that for L ^ B 

/ fa {mlm~^) dm = 0, 

J M/L 



hence 



tr7r(^g.) = iT^/pM / / faimbm ^) dm Q^{ab)d{ab) db. 
\'^\^)\ Ja+bJm/b 



The character of vr is non-zero only on Cartan subgroups of G that are G-conjugate to Cartan 
subgroups of MA (see [26], Proposition 10.19). The subgroup BA is not G-conjugate to any Cartan 
subgroup of MA, so it follows that tr7r($(^) = 0. □ 

Now let TT = Ind p(^ (8 i') for some ^ G M and v G a*. For r G Km let Pt : V^{t) be the projection 

onto the r-isotype. For any function f on G which is sufficiently smooth and of sufficient decay the 
operator 7r(/) is of trace class. Its trace is 

V / / a^+P"" f{k-'^mank) tvPr^{m)Pr dman dk. 
— Jk J man 

tEKm 

Plugging in the test function / = where a G {triv, a}, this gives us, as in [9]: 



tr7r($,)= / G{a) tr 
Ja+ 



da. 



where C(a) depends only on a and is an Euler-Poincare function on M attached to the representation 

a. We can sec that tr'iT{^a) is non-zero only if ti ^{fa) is also. 

Theorem 2.11 now follows from Proposition 1.12. 



3 A Prime Geodesic Theorem for SL4(M) 

We continue using the notation defined in the previous sections, in particular wc take G = SL4(R.) and 
take F C G to be a discrete, cocompact subgroup. For 7 e F let A^(7) — e^'' and define for x > 

^{x) = Xii^j), and 7f(a;) = ^ xi(r7)tr ct(6^), 
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where F-y is the centrahser of 7 in F, we denote by Xi(r7) the first higher Euler characteristic and a is 
the virtual representation defined in Section 2.4. We recall from Proposition 1.10 that the first higher 
Euler characteristics are all positive and from Lemma 2.10 that the traces trcr(6^) are also all positive, 
so both -Kix) and t^{x) are monotonically increasing functions. 

Theorem 3.1 (Prime Geodesic Theorem) For x ^ 00 we have 

7r(x) ~ and Tt{x) 



log X log X 

More sharply, 



2,3/4 \ / ^3/4 



7r(a;) = 2]i{x) + O and tt{x) = 8li{x) + O 



^ log X J \ log x 

as X —> 00, where \i{x) = 15^7^* integral logarithm. 

We also prove the following Prime Geodesic Theorem, which will be needed for our application to class 
numbers. 

Let B° be a closed subset of the compact group B with the following properties: it is of measure zero; 
it is invariant under the map b and contains all fixed points of this map; and its complement 

= B \ B'^ in B is homeomorphic to an open subset of Euclidean space each of whose connected 
components is contractible. The assumption that B^ contains all fixed points of the map & 1— > is 
equivalent to the assumption that B'^ contains all non-regular elements of B. Let (fp ^(F) be the subset 
of all [7] G <op{T) such that 7 is conjugate in G to an element of A~B^. We define for x > 

n\x)= ^ xi(r^). 

N(y)<x 

Then as in the case of 'jt{x) above, 'jt^{x) is a monotonically increasing function. 
Theorem 3.2 For x ^ 00 we have 

TT [X) 



logx 

The proof of these two theorems will occupy the rest of the section. 



3.1 Analytic properties of Rr,a{s) 

This subsection and the following proceed according to the methods of [34] and [35], in which the Selberg 
zeta function for quotients of the hyperbolic plane are considered. In this section the analogs of a series of 
lemmas are proved in our context. The next section translates the main theorem of [34] into our context. 

Recall from Theorem 2.8 that for a finite dimensional virtual representation cr of M we have the functional 

equation 

Zp,^{l -s)= e-«(«)Zp,<,(s), (38) 
where G{s) is a polynomial. Let D be the degree of the polynomial G{x). 

Lemma 3.3 Let H be a half-plane of the form {Re(s) < —(1 + e)} for some e > and let a be a finite 
dimensional virtual representation of M. Then there exists a constant C > such that for s G H we 
have 



\R'r,As)/RrA-^)\<C\ 



s 



\D-1 



Proof: From Theorem 2.9 we get the identity 

4 

9=0 

which implies 
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Considering this identity, it will suffice to prove that when is a half-plane of the form {Re (s) < —s} 
for some e > 0, there exists a constant C > such that for s e if we have 



l-^p,(A''n®^<.)(*)/'^^>(A''n®v-<,)(s)l < G\s\ 



D-l 



for all g = 0, . . . , 4. Since the proof does not depend on the value of g or on cr we shall abbreviate our 
notation for the zeta function to Zp{s), which notation we shall use for the rest of the section. 

It follows that 

Z'pjl - s) _ Z'pjs) 
Zp{l-s) Zp{s) ^ 

From the definition (16) of Zp{s) and Proposition 2.12, we can see that Zp{s) is both bounded above 
and bounded away from zero on the half plane K' = {Re(s) > 1 + £}. It follows that Z'p{s)/Zp{s) is 
bounded on K' . This implies the lemma. □ 

For t > 0, let N{t), denote the number of poles and zeros of Zp{s) at points s = \ +x, where Q <x <t. 

Lemma 3.4 N{t) = 0{t^) 

Proof: Define ^(s) = {Zp{s))^ e~^^^\ where G(s) is the polynomial in the functional equation (38) for 
Zp{s). We note that, in light of its role in the functional equation, the polynomial G{s) must satisfy 
G{s) = -G(l - s). It then follows that 

C(l-s)=^(s). 
Note that ^(s) is real on the real axis and so ^(s) = ^(s). 

Fix a real number 1 < a < 5/4 and let t > 0. Let R be the rectangle defined by the inequalities 
1 — a<Res<a and —t<\ma<t. We assume that t has been chosen so that no zero or pole occurs 
on the boundary of R. Then 

^' 4 2ni Jgji e(s) 4 27r Viaii ^(s) J 

where A^o is the number of poles and zeros of Zp{s) on the real line. It then follows from the functional 
equation for ^(s) and from the fact that ^(s) = ^(s), that we have 



No, 



where C is the portion of dR consisting of the vertical segment from a to a + it plus the horizontal 
segment from a + it to ^ + it. 

Now the definition of ^(s) gives us 



2^-G'(s), 



so that 



as) zp{s) 



Im i I ^ds] = 2.1m ( f ^ds] -Im 



m J \Jc zp{s) 



[ G'{s)di 
Jc 



It thus remains to show that 



= ^■'°'(Xifw'"')-'°'(°G+")^°<''> 



Note that S(t) is the variation of the argument of Zp{s) along C. We may extend the definition of S{t) to 
those values of t for which is a pole or zero oiZp{s) by defining it to be limg^o \ {S{t+e) + S{t — e)). 

Prom the definition of Zp{s) we can see that S(t) = h{t) + 0{1), where h(t) is the variation of the 
argument of Zp{s) along the segment from a + it to ^ + it. The value of h{t) is bounded by a multiple 
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of the number of zeros of Re(Zp(.s)) on this segment, since the point Zp{s) cannot move between the 
right and left half-planes without crossing the imaginary axis. On the segment, the real part of Zp{s) 
coincides with 

fp{w) = ^{Zp{w + it) + Zp{w - it)), 

where w runs from ^ to a on the real axis. Since we have assumed that + it is not a zciro or pole of 
Zp{s), the function fp{w) is holomorphic in a neighbourhood of the closed disc S, centred at a, of radius 
a — 5. As this disc contains the interval from | to a, we may apply Jensen's Formula ([6], XI. 1.2) to 
conclude that 



h{t) = O (^J^^\og\fp{w)\dw 

= oil \og\Zp{w + it) + Zp{w-it)\d'. 

\JdS 

= o(^J \og\Zp{w + it)\+log\Zp{w-it)\dw 

= O V/ \og\Zi{w + it)\dw+ [ log\Zi{ 
\i=i,2''9S J as 



w — it)\d'w 



There remains one final step in the proof of the lemma: 

|Zi(a; + it)| =e°(l*l°) (40) 

uniformly in the strip 1 — a < a; < a for i = 1, 2. We know that the function Zp{s) is bounded and 
bounded away from zero on the line Res = a. Hence, by Lemma 2.6 and the functional equation (38), 
we can apply the Phragmen Lindelof Theorem ([6], Theorem VI. 4.1) to give (40). □ 

Lemma 3.5 Given a < 6 G M, there exists a sequence {yn) tending to infinity such that 



R'Ax + iyn) 



= 0{yl^) 



Rr{x + iyn) 
for a < X <b. 

Proof: As in Lemma 3.3 it will suffice to prove the result for Zp{s). 
Using the notation of Lemma 2.6 we have that 

^ hn,-n,)+g[{s)-g',{s)+Y.{-iy-' s'p-^is-p)-\ 

Let to > 2 be fixed and consider the segment of the line Re s = ^ given by i + it for t^ — 1 < t < t^ + 1. 
Let N{t) be as above, then by Lemma 3.4 we know that N(t) = 0{t^). It follows immediately that the 
number of roots on the segment is Oit^). 

By the Dirichlet principle, there exists a i + iy in the segment whose distance from any pole or zero is 
greater that C/T^, for some fixed C > 0. We conclude that the portion of the sum 

Y^{-iy-^Y.^''p-\s-p)-^ 

i=l,2 p 

corresponding to poles and zeros in the segment for Sx = x + iy'-is 0{y'^^), since |s^p~'°| = 0(1) for these 
p, when a < X <b. 

To deal with the segments | + it for < t < to — 1 and to + 1 < t < 00, we proceed as follows. The 
portions of the sum X]i=i 2(~-'-)'~^ J2p s''p~''{s — p)~^ corresponding to the first and second segments 
respectively, can be written 

and 

Recalling that N{t) = 0{t^), we easily conclude that both of these expressions are 0{T^^). □ 
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3.2 Estimating 'ip{x) and tlj{x) 

To simplify notation, in what follows we write 7 for an element of ^p(r) and 70 for a primitive element 
and recall that 7 (E 'S'p{r) implies that 7 is conjugate in T to an element 0,-,,^. G A~B. Unless otherwise 
specified, sums involving 7 or 70 will be taken over conjugacy classes in ^p{T) and S'p(r) respectively. If 
7 and 7o occur in the same formula it is understood that 70 wil be the primitive element underlying 7. 
We denote by S'p^{T) the subset of regular elements in ^p(r). For a; > let 

ble^pCr) 

N(-,)<x 

and 

Let C7 be a finite dimensional virtual representation of M. We have for Re(s) > 1: 



Exi(r^)tra(6^)/^„e-^'-. (41) 



The following propositions are analogs of Theorem 2 of [34] , from which, in the next subsection, we prove 
the Prime Geodesic Theorem using standard techniques of analytic number theory. 

Proposition 3.6 tp{x) = 2x + O (x^/^) 

Proof: Let D be the degree of the polynomial G{s), as in the previous section, and suppose k > 2D is 
an integer and x,c> 1. Then, by (41) and [18], Theorem 40, 

— / J ) s-\s + 1)-^ ■ ■ ■ (5 + ky^x'ds 

= . (E^i(r7)^7ce-^'^J s-^(s+l)-i---(s + fc)-ix^ds 

= h. E »(r,)..(.-^)'. 



(42) 



JV(7)<x 



Theorem 2.11 tells us that all poles of Ry{s)/Rt{s) lie in the strip — 1 < Re(s) < 1. By virtue of 
Lemmas 3.3 and 3.5 it is permissible to shift the line of integration in (42) into the half plane Re(s) < — 1, 
taking into account the residues of the poles of iip(s)/i?r(s)- Hence, for c' < — 1 



^Ex.(r*.(.-^)' 

27^^^'-icx> -«r,i(s) 



(43) 



Re (a)>c' 

where denotes the set of poles of (i?p j(s)/iir,i(s))s~^(s + 1)~^ •••(« + fc)"^ and Cfe(Q!) denotes the 
residue at a. 

For a; > 1, Lemma 3.3 implies the integral in (43) tends to zero as c' — > —00. If we define 

'^q{x)=i\)(x), iPj{x)= [ 'il}j-i{t)dt, j gN, 
Jo 

it is well known that 

Nij)<x 
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and we deduce from (43) that 

Mx) = Cfc(a):r'=+". (44) 
Let d > 0. For a function / : M — > M define the operator A by setting 

2D 



/2D\ 



=0 

It follows from (44) and Theorem 2.11, setting k = 2D, that 

^2i.(x) = — ^^^^+^+ C2D(a)x2^+"+ E cMa)x'^^'^, (45) 

where S'^q = S2D fl (M \ {!}), the real elements of S2D not including a = 1, and S'l^, = S2D H (g + 
i(M \ {0})), the non-real elements of S2D on the line Re(s) = q. The coefficient ((2D + 1)!)~^ on the 
leading tc;rm clonics from the fact that (iZp i{s)/Rr,i{s)) has a double pole at s = 1 and from the factors 

s-\..{s + 2D)-\ 

In general if / is at least 2D times difFcrcntiablc, 

/■x+d /•t2n+d ft2+d 

A/(a;)=/ / •••/ f^^'^Hh) dh...dt2D. (46) 

By applying the Mean Value Theorem we get 

Ax'' = d^^r{r - l)...(r - {2D - l))x'-^^ , (47) 
where x & [x,x + 2Dd\. In particular we notice that A(a;^^+^) = 0{x), hence 

^ 2Z3+l\ ^^^^ 



,(2£> + l)! 

for some a, 6 e M. Computations show that 

2D 

^ = 2E(-1)' •!(^^— jT((2^ - = '^d'''- (48) 



By definition ipoix) = '>P^d\s) so from (46) we have 



'^2D 

Ai,2D{x)= / ••• / Mh) dti...dhD- (49) 

Jx Jt2D Jt2 

By Proposition 1.10, the Euler characteristics Xi(r7) are all positive. Hence ij}o{x) is non-decreasing and 
it follows from (49) that 

^^0(2;) < d-"^^ A^I)2d{x) < Mx + '2Dd). (50) 
It also follows from (47) and (48) that 

rf-^^A I — ^x^^+i + E C2D{a)x'^^-]=2x + 0{x'/'). 

Thus it remains to show that for p G {—2,-1, 0, 1, 2} 

/ 



d 



-2D, 



E C2,,(a)x2^+" I =0(^3/4) 



and the proposition follows by (50). 
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Let p e {—2,-1, 0, 1, 2}. In order to estimate 



we need two estimates for A {c2d{o()x'^^~^"), where a e Sj^^. The residues at the poles of iip i{s)/Rt,i{s) 
are 0(1), so for a e S^(^ 

rf-2^A (c2Z5(a)x2^+") = O (d-2^|a|-(2^+i)a;2^+f/4) . 

On the other hand, it follows from (47) that 

rf-2^A {c2D{a)x^''+") = O {\a\-^xP/^) . 

Define Np{t) to be the number of poles of R'p ^{s)/Rr,i{s) on the interval | + i{0,t]. Prom Lemma 3.4 
we have that Np{t) = O (t^). Thus 



-2D , 



\ 



C2D{a)x 



2D+a 



(51) 



If we choose K and d appropriately, then we can conclude from (51) that d "^A (j2^^gp/4 C2(a)a;^+"^ 
0(a;'^/''), as required. 

Proposition 3.7 ip{x) =8x + (x^^^) 



□ 



Proof: The proof follows exactly as for the previous proposition, replacing iir,i(s) with Rr,a{s) through- 
out. It follows in particular from the fact (Theorem 2.11) that Rr.a{s) has a zero of order eight at the 
point s = 1 and from the fact (Lemma 2.10) that tra(6^) > for all 7 G <?p(r) and tra{bj) = if and 
only if 7 is non-regular. □ 



3.3 The Wiener-Ikehara Theorem 

We shall use the following version of the Wiener-Ikehara theorem (see also [5], Chapter XI, Theorem 2, 
and [10], Theorem 3.2). 

Let i?/c(s), fc S N be a sequence of rational functions on C. for an open set U C C let N{U) be the set of 
natural numbers k such that the pole divisor of Rk docs not intersect U. We say that the series 

feeN 

converges weakly locally uniformly on C if for every open U cC the series 
converges locally uniformly on U. 

Theorem 3.8 (Wiener-Ikehara) Let A{x) > be a monotonic measurable function on M+. Suppose 
that the integral 

/•oo 

L{s) = / A{x)e-''' dx 
Jo 
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converges for s G C with Rc(s) > 1. Suppose further that there are j G N, r G M and a countable set I, 
and for each i € I there is 9i € C with Ke{Oi) < 1 and Ci G Z, such that the function 




extends to a holomorphic function on the half-plane Re (s) > 1 . Here we assume the sum converges weakly 
locally uniformly absolutely on C. Then 

lim A(a;)a;~^-'+^)e~^ = r. 

x—^oo 

Proof: This is a straightforward variation of the proof of the Wiener-Ikehara-Theorem in [5] . □ 



3.4 The Dirichlet series 

Let be a closed subset of the compact group B with the following properties: it is of measure zero; 
it is invariant under the map 6 i— > and contains all fixed points of this map; and its complement 
B^ = B \ B'^ m B is homeomorphic to an open subset of Euclidean space each of whose connected 

components is contractible. 

The Weyl group W = W{M, B) contains two elements and the non-trivial element acts on B by 6 i— > b~^, 
so the invariance condition above says that both B^ and B^ are invariant under the action of W . The 
fixed points under the action of the Weyl group are precisely the non-regular elements of i?, so the 
assumption that B^ contains all these fixed points is equivalent to B^^^^ c B^, where B^^^^ denotes the 
subset of nonregular elements of B. The action of W on B^ permutes the connected components and the 
assumption c -B° implies that the quotient space B^ /W is also homeomorphic to an open subset 

of Euclidean space each of whose connected components is simply connected. 

For subsets S and T of G we denote by S.T the subset 

S.T = {sts-^ ■.seS,teT} 

of G. Let Mg\i = M.B^ c M. Let ^^{T) and S'^{T) be the subsets of S'p{T) consisting of all conjugacy 
classes [7] such that 6-y e B^ or G B^ respectively. The assumption that B^ is of measure zero is not 
required for the following lemma, but will be needed later on. 

Lemma 3.9 There exist a set C M^^jj with compact closure in M and a monotonically increasing 
sequence ((;„) of smooth, functions on M , supported on M^, which are invariant under conjugation by 
elements of Km, and whose orbital integrals satisfy 

C'af (Sn) = / gn{xbjX~'^) dx^l as 00 

J M/B 

for all -ye <^^(r). 

Proof: We view M^jj as a fibre bundle with base space B^/W and fibres homeomorphic to M/B. 

Let d{-,-) denote the metric on B given by the form b in (12). For n G N let -B„ c B^ be the set 
Ba ^ {b e B : d{b,B") > l/n} and let B„ Bn/W. Then, by [41], Corollary 1.11, for each n G N 
there exists a function /i„ : B/W R such that < /i„(6) < 1 for all b G B/W, for all 6 G we have 
hn{b) = 1 and /i„ is supported on B^/W. We may assume that the /i„'s form an increasing series. 

Let fJ be a compact neighbourhood of a point in M/B homeomorphic to a subset of Euclidean space. 
Let k : M/B — > K be a smooth positive function supported on U and satisfying J^/^ k{m)dm = 1. 

We now define, for n G N, functions g„ : M ^ R as follows. On each connected component V of B^ /W 
we fix a trivialisation of the restriction to V of the bundle M/B M^^ B^ /W . Then for v &V and 
m G M/B define gn{mvm,~^) = hn{v)k{m). Since V is simply connected there are no problems with 
global agreement of this definition. For m G M \ M^jj define gn{fn) = 0. Finally we define the functions 

5„ : M ^ M by 

9n{m) = 77 / gn{kmk~^)dk 

^ J Km 
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for all m G M. Then the functions g„ form an increasing sequence of smooth i^M-iiivariant functions 
supported on the compact set Mg, which we define to be the closure in M of Km-{U.B^). We will show 
that their orbital integrals have the required properties. Let b G B^. 



0^{gn) = I gn{mbm ^) dm 
Jm/b 

= \ I I gn{kmbm~^ k~^) dkdm 
2 Jm/b J Km 

= 7: I I gn{fnkbk~^m~^) dkdm 
2 Jm/b J Km 

~ \ l f gn{inbm~^) + gn{mb~^m~^) dkdm 
2 Jm/b Jb 

= \{K{b) + hn{b-^)) k{m)dm 
^ Jm/b 

= K{b). 



The last equality holds since bW = 6 in B'' /W . There exists e N such that b e Bn for ah n> N, 
hence 0^{gn) — > 1 as n — > 00, by the definition of the functions /i„. □ 



For n e N let 



let 



and let 



[7]e<?i.(r) ^ V 7 7/ I / 

rn= gn{x)dx, 

Jm 



mn,x = E(-l)'^» dimif«(n, t^k)^. 

g=0 



Proposition 3.10 For alln gN and for j € N large enough the series Li^{s) converges locally uniformly 
in the set {s e C : Re(s) > 1}. 

The function Lj^{s) can be written as a Mittag-Leffler series 

where the summand of the double series corresponding to tt = triv, A = —2pp is excluded. The double 
series converges weakly locally uniformly on C. For n G G, X G a* such that (tt, A) ^ {triv,—2pp) we 
have m-n.x ^ only ifRe{X) > —2pp. Thus, in particular, the double series converges locally uniformly 
on{sGC: Re(s) > 1}. 



Proof: Wc can put / = g„ in (14) to define the function which by Proposition 2.3, for j and 

Re (s) large enough, goes into the Selberg trace formula to give the equation 

E^rWtr.($,J= E vol(rAG,)a. 1;-;;^^ 

By the definition of the functions the orbital integrals Ob^{gn) are equal to zero when 7 e S'p{r). 
Also, for 7 <E ^op(r), since we have assumed that B'"''*s c B^ we have that 7 is regular and we get from 
Lemma 1.9 that 

voi(r^\G^) = xi(r^)«7o- 

Thus we can see that the geometric side of the trace formula is equal to l4i{s). It follows that for j and 
Re(s) large enough the Dirichlet series L^{s) converges absolutely. 
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Lemma 3.11 



Ago* 



Proof: Replacing with (?„ in (22) we get 

tr7r($g„) = / g„{m)Q^.'\^^^^{ma)dmgi{a)da 

Jma- 

= I I gn{m)dm tr{a\H*{n,nK))li'^'^e-^^''da. 

J A- Jm 

/•oo 

= r„ / V diniif(n,7rK)^e(^(-f^i)-*)*i^+idi 



□ 



By the above lemma and Proposition 2.15, and using similar arguments to those used in the proof of 
Proposition 2.17, we can see that the spectral side of the trace formula is equal to the Mittag-Leffler 
series given in the propostion. 

Since ^^(s) is a Dirichlet series with positive coefficients it will converge locally uniformly for s in some 

open set. By proving the convergence of the Mittag-LcfFlcr scries wc shall show that the Dirichlet scries 
extends to a holomorphic function on {Re(s) > 1} and hence, since it has positive coefficients, converges 
locally uniformly there. 

We recall that for A e a* we write ||A|| for the norm given by the form b in (12). It then follows from 
Proposition 2.5 that there exist mi e N and C > such that for every n G G and every A € a* we have 

|m„,A| <C(1 + ||A||)™^ 

If S denotes the set of all pairs (tt. A) G G x o* such that m„,A 0) then there exists m2 € N such that 

iVr(7r) 



(7r,A)eS 



(1 + 



< OO. 



Now, let [/ C C be open and let S{U) be the set of all pairs (tt. A) € G x o* such that mn,\ and 
A(i?i) ^ U. Let y be a compact subset of U. We have to show that for some j € N, which does not 
depend on U or V, 



sup 

sev 



E 

(7i-,A)eS(i7) 



7Vr(7r)m„,A 



< oo. 



Let nil, 7712 be as above and let j > mi + m2 — 2. Since V d U and V is compact there is e > such 
that s e y and (tt. A) G S{U) implies |s — A(iJi)| > e. Hence there is c > such that for every s G V 
and every (tt. A) S S{U), 

\{s - X{Hi)\ > c{l + 



Putting this all together we get 



sup 
sev 



E 



(7r,A)eS((7) 



Nr{n)mn,} 



1 A^r(7r)|mn,A| 

- -'(.,Afe(.)^^"^(l + ll^ll)^'"^ 

>^ C Nrjn) 

^ E ^(T^ 

(7r,A)eS(C/) ^ 

< OO, 



which proves the proposition. 



□ 
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3.5 Estimating ipnix) 
Let 



]V(^)<x 

Lemma 3.12 

Proof: This follows from Abel's summation formula ([5], Chapter VII, Theorem 6). □ 
Let 

N(l)<!C 

Lemma 3.13 For each n gN we have 

(l>nix) ~ rnX. 

Proof: By Propostion 3.10 and Lemma 3.12 we can apply Theorem 3.8 to the series Lj^{s) and the 
function to deduce that 

lua^p^^=rr.. (52) 

Also it is clear that 



(logx)J+i' 
so it follows that 

limmt — ^-^ > r„. 

X— ^oo X 

Let < M < 1. Then 

a>C<N(7)<x 



> 



xf <JV(7)<x 



Prom this we get 



M^) ^-(.7 + 1) '^n(^) I MX^) /gg-. 



Assume first that <pn{x)/x tends to infinity as a; ^ oo. Then there is a sequence Xm of positive real 
numbers, tending to infinity such that (p{xm)/xm tends to infinity as m ^ oo and 

for all m. Then 

.-U+l)_ KiXm) , 4>n{Xrn) 1 



Xm Xm 

SO that 
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By (52) the right hand side converges, so we have a contradiction. This impUes that Hnia;^oo '^"j^^ exists 
and is finite and we set 

r y 4>n{xi^) 
L = limsup = limsup . 

X— ^oo X x—^oo X^ 

From (52) and (53) we get 



L < r„At-(^+^) +iHnisup^— = r„/i-(^+^). 
Since this holds for any value of /z in the interval < /x < 1 we get that L <r and the lemma follows. 

□ 

For n e N and a; > let 



Proposition 3.14 For each n e N we have il>n{x) ^ r„a;. 

Proof: We note that < det (1 — ci^&^Iti) < 1 for all ^ € (ap *^(r) and that the value of the determinant 
tends to 1 as tends to infinity, hence for < e < 1 there are only finitely many 7 e S'p^{T) such that 
det (1 - a^fe^ln) < 1 - e. Since ^^(F) c <fp «(F) the same holds for 7 € <fi>(F). We fix < e < 1 and 
define for each n € N the functions ^n,s and ipn,s to be the same sums as for ^„ and ipn respectively but 
restricted to those classes [7] e <^p(r) such that 1 — e < det (1 — a^b^jn) < 1. It then follows that both 

(f>n{x) - ^nA^) _^ (54) 



and 

as a; — > 00. Now 
immediately implies 

so summing up we get 



1 — £ < det (1 — ajbj\n) < 1 



1 - r 1 
< 1 < 



X 

By Lemma 3.13 and (54), it then follows that 



det (1 — a^b-yln) det (1 — a^b^ln) ' 

<Pn,e{x) _ ^-j ^ i^n,e{x) ^ (l)n,s{x) 



r„(l — e) < iimmi < limsup < r„. 

X ^00 X X — ^00 X 

It then follows from (55) that 

/, ^ / r • f ^ni^) / r V'«(a;) ^ 
r„(l — e) < limmt < limsup < r„. 

^ X X — ^00 X 

Since this holds for any value of s in the interval < £ < 1 the proposition is proven. □ 
3.6 Estimating ^^(x) 

Lemma 3.15 The sequence (r„) is monotonically increasing and r„ — > 2 as n —> 00. 

Proof: That the sequence (r„) is monotonically increasing is clear since the sequence of functions (c/„) 
is monotonically increasing. 
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By Wcyl's integral formula ([26], Proposition 5.27), and since the functions Qn are only non-zero on elliptic 
elements of M, we have 



/ gn{x) dx 

JM 

0^{9^)\D{h)\^ db, 



\W{B:G)\ 

where W{B : G) is the Weyl group and D(b) is the Weyl denominator. Since the sequence of functions 
Qn is monotonically increasing and the functions are all supported within a given compact subset of M 
we can interchange integral and limit to get 



Furthermore, the orbital integrals O^^ {gn) tend to one as n ^ oo, except for 6 in a set of measure zero, 
so it follows that 



The Weyl group W{B : G) consists of the identity element and the element diag(l, — 1, 1, —1). We can 
also compute the Weyl denominator D{h) for b G B. Let 

i?(^)=f e S0(2) 

^ ' ' sm^ cos^ ' ^ ' 



and let 



Then 



D{R{e,<t>)) = e*(«+'!i)(l_e-2»^)(l-e-2*'^) 

= 2cos {e + (/))- 2cos {e - 4>) 

= 2 (cos ^cos (p — sin ^sin <f) — cos ^cos <f) — sin 0sin (j)) 

= — 4sin0sin(i 



We have normalised the Haar measure on B so that Jgdb = 1, hence 



2 , d6 d(j) 

sm 



2 p2Tr r2ix 



^ JO JO 
= — ^.TT = 2. 

This proves the lemma. □ 
Let 



Proposition 3.16 ■ip'^{x) ~ 
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Proof: For all n G N and all .t > wc have ^l>n{x) < ip^{x) < ip{x). It then follows, using Proposition 

3.6 and Proposition 3.14, that for all n € N 

Tn = hmmi < limmi < limsup < limsup = 2. 

X ^oo X X ^oo X X — *oo X X — *oo X 

We can then deduce from Lemma 3.15 that lim^^^tx) ^ =2. □ 

3.7 Estimating ti{x), t:{x) and 7r-^(x) 

To keep the notation less cluttered, in the sums over conjugacy classes in F which appear in this section 
we shall not specify which set of classes the sum is being taken over, since this will be clear from the 
context. We shall always use 70 to denote primitive elements and where 7 and 70 appear together in the 
same formula we shall mean that 70 is the primitive element underlying 7. 



hm = hm ^ = 1. 



Proposition 3.17 

+oo2a;/log3; x^<x 2x 

Proof: We can write 



^'(2;) = XI "7oXl(r7o)^7oi 
JV(7o)<x 

where n-y^ e N is maximal such that N{'yo)""^° < x. By definition A'' (70) = e'i'o , so A''(7o)" < x implies 
nl-yg < log X and we can see that 

ipix) < log(a;)7r(a;). (56) 

Next wc fix a real number < a < 1. By Proposition 1.10 the Euler characteristic Xi(r'7o) > for all 
70 G ^^(r), so for x>l, 

i^{x) > X xi (1^70)^70 • 

x»<JV(7o)<x 

As above, N{'yo) > a;" implies l^g > logo;", hence 

il){x)>alogx X] Xi(r7o) = alogx(7r(a;) - 7r(a;")). 

x'»<JV(7o)<x 

Since F c G is discrete, Tr{x°') < Cx"" for some constant C so 

V'(a;) > a7r(a;) logx — aCa;" logx, 

which gives 

il){x) , Aogx logx 
> a'iT[x)— aC- 



2x ' ' 2x 2a;i-«' 

Since < a < 1, it follows that (loga;)/a;^~" as a; ^ 00 and 

liniM>alim 



x^oo 2a; x^oo2x/loga; 
for all < a < 1. Hence 

lini M > lim 



+00 2a; a;^(x) 2a;/ loga; 

Together with (56) and Proposition 3.6 this proves the proposition. □ 



Proposition 3.18 



lim "/f) = lim ^ = 1. 
x^oo 8a;/ loga; x^oo 8a; 



Proof: Exactly as for the previous proposition, making use of Proposition 3.7 and Lemma 2.10. □ 
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Proposition 3.19 



Proof: We consider the function 



wix) = 2li{x) + O 



x^ 
logx 



s{x) = ^ xi(r^o)^ = E xi(r,o) + E E xi(r^o)^- 



]V(7)<x 



JV(7o)<x 



fe>2 JV(7o)<cci/fe 



We consider the double sum on the right. Since F C G is discrete there is a geodesic 7niin of minimum 
length. For a given x the inner sum contains at least one summand only for k < \ogx/l^_^^_^. For each 
such A; > 2, by Proposition 3.17, the inner sum is equal to 0{-\/x/\ogx). Therefore we have 



S{x) = 7t{x) + O {Vx) . 



Now 



rM-dt = r xr(r7o)-^ 

J 2 tlog t J 2 „77f^, Hog t 



N('y)<t 



dt 



E r xi(r7o) 



ilog^t 



dt 



N{j) 



= S{x) 



h)<x 



logo; 



Hence ^(a;) = tto^t lo^' Proposition 3.6 we have 

s{x) = r 7^dt+-^+o( r —^dt)+o( 

J 2 \og^t log a; Vi2 ii/^log^t / V 

r — 

J 2 logi 



x^ 
logx 



\ogt 

2 logt 



dt + 



dt + 



logx J 



2x 
logo; 



logo; 



Together with (57) this proves the proposition. 
Proposition 3.20 



Tr{x) = 8li{x) + O 



x^ 
logx 



(57) 



□ 



Proof: Exactly as for the previous proposition, making use of Proposition 3.7 and Proposition 3.18. 



□ 



The proof of Theorem 3.2 proceeds exactly as for the proof of Proposition 3.17, making use of Proposition 
3.16. 
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